Andrew Wilkinson

LDX3 London 2025 (Day 2)

2025-10-24T00:00:00+00:00

This is the second part of my write up of the LeadDev London 2025 conference. To read my thoughts on the talks from day 1, and my overall impression of the conference please read my previous post. As with that post this is just the summary of my notes, so will give you a flavour of the talks.

I attended a training session with Suzan Bond in the morning. I’ve included my thoughts on that at the end of this post.

The videos of the talks are available here if you sign up with a free LeadDev account.

From dashboard soup to observability lasagna: Building better layers

Martha Lambert (Video)

Designing for reliability is made up of two parts - proactive and reactive. Proactive work is knowing that your system will be ok, while reactive is knowing that you will be alerted when it’s not. If you just start building dashboard, metrics, etc, then you’ll end up with soup. Instead you need to design them.

Start with a prediction - how should the system work?

Prove that prediction - show what it actually does.

Then measure - what can’t you answer with the data you have?

Predictions can’t be tested with tests. You need experience of what strain looks like, e.g. alert storms.

Typically, dashboards only answer specific, often now irrelevant, questions. They can be static and disconnected from the rest of your stack. You should consider dashboards as part of your product.

Connect layers of observability to enable you to dig into problems. Start with overview dashboards, link them to system dashboards, which link to logs, and finally link to traces. The overview should be a triage tool to show you the infrastructure health. Systems dashboard should show the complete health of the system.

Always think about user impact. Don’t think about how long does it take? Think about how long does it look to the user?

Growing pains: Scaling and re-architecting systems under fire

Vicki Kalmanovich (Video)

The first step when you are facing growing pains is to get buy-in on the pains, the solution and the process. The pains you are facing could be availability, scalability or reliability. Talk about the impact and the cost proposition.

For solutions it helps to get buy-in from stakeholders, even if it’s an engineering decision.

Level of effort + impact = return in investment

You can’t just disappear during the project, you need to get buy-in on the process so you can keep getting the buy-in to continue. As part of the process, think about your exit strategy - think about how and when to change tack.

Being secure by design: Engineer-led security

Dan Abel (Video)

Security is a serious business, and the risks are high. The risk to your company’s reputation, operation and data. If you don’t take security seriously there is a risk to your teams, even without an incident - your company may hire external consultants and introduce process and red tape.

DevSecOps is the movement to include security in DevOps and make security everyone’s problem.

To introduce this to his company Dan ran workshops with all stakeholders to find out requirements. They shared information to show people the secure path, and gave them support. They taught devs how to think about risks.

They also collected data and reported to the Head of Department about vulnerabilities in third party libraries. They categorised the data based on a risk and provided a path for handling it.

Dan finished by saying that security threats will only increase, but engineers optimise themselves when you give them a mission.

Balancing direction and empowerment

Lara Hogan (Video)

Lara Hogan is a LeadDev regular, and in this talk she discusses how to balance telling people what needs to be done against giving people freedom to choose their own path. She starts by reminding us that none of the engineering management practices we spend so long talking about matter, if the business doesn’t succeed.

Consider what you are empowering people towards, if the business is faulty? Are you optimising to minimize layoffs? Efficiency? This is your “why”.

There are two ends to the management spectrum:

Empowerment - Used when you need to gather buy-in, when you need creative ideas, and when you have lots of time.
Direction - Used when people need clarity, when you have new people, when there are big risks to the company and when you have no time.

Pay attention to the context and be intentional about where you are on this spectrum.

Neurodiversity: From struggles to solutions, tips for leading a team

Stacey Nutbrown (Video)

This is a nice simple talk organised around eight tips for leading neurodiverse people. The talk starts by discussing the difference between authoritarian, permissive and authoritative leadership styles. It then features several tips about how to approach difficult conversations, and how to work with your reports to come up with solutions to problems together.

What’s my job again? Developing self-management

Cate Huston (Video)

This is a talk about how to work out what your job is, and to come up with a strategy for doing your best work, all through the metaphor of racoons.

It is important to remember that good strategy is only good in context. When the context changes, you need to change your strategy. Cate talks a bit about product strategy, a bit about technical strategy, and a bit about how you can use different leadership styles to get things done in different contexts.

So you want to hire engineering force multipliers?

Hazel Weakly (Video)

If you want to hire force multipliers, you’re looking for unusual people. Hazel’s talk attempts to show you how to find them by experimenting with your interview process and assessing it for bias.

You should be looking for people who…

Build a cumulative culture.
Have intellectual humility - people need to like you to be a force multiplier.
Ecological awe - they should not optimise for one thing, they look for a balanced ecosystem.

Use questions that can’t be answered to filter out people trying to fake it with AI. Test your questions extensively, including on your own team.

As an example Hazel considered the question “How would you approach splitting a monolith?” Instead they propose replacing it with “Let’s work together and examine the process of splitting a monolith. How would you approach this culturally and technically? Let’s focus on the human elements first.”

Getting excited about maintaining legacy systems

Blanca Garcia Gil (Video)

Maintaining software is a different skill to working on a greenfield project. Legacy code is part of every job, but it provides an opportunity for learning - there are many stories encoded in the code.

Tech debt is linked to developer productivity and happiness. As senior engineers we lead the way for others, and set the tone for dealing with legacy code.

Doing nothing to fix technical debt results in the “software event horizon”, where it can become impossible to fix bugs. In this talk Blanca discusses how to use systems thinking to create a map that will help you to understand what the code is supposed to do. Talk to people involved in the system. Look at the edges to see what it is connected to.

Document incident reviews, track where technical debt might be introduced into your system, and monitor the developer experience. Use these to develop practices to prevent the technical debt from growing.

A safe place to land: Practical (and hyper-scale tested) advice for technical decision-making while evolving systems in place

Inés Sombra (Video)

In this talk Inés discusses how to create a process for making technical decisions while working on an existing system. In it she talks about how Fastly evolved their systems dashboard, and the process made it more reliable, easier to use and easier to innovate with. She also discusses how they used AI to help make decisions, and speed up the implementation of their changes.

Mastering Managing, Mentoring, Coaching (Workshop)

Suzan Bond]

As usual at LeadDev conferences they offer a series of microworkshops alongside the standard talks. I attended the session on mentoring and coaching with Suzan Bond. This was a good session, with a nice mix of presentation, discussion and pair work. It was definitely a nice change from sitting listening to talks all day.

In the session Suzan began by discussing the difference between mentoring and coaching. Mentoring is telling someone, while coaching is asking questions to help them find their own answers. Mentoring is easy, because you just tell people what you did and what your experiences were. Coaching is harder.

You should design the alliance between the two of you. How do you want to work together? Try to create a partnership. When discussing things “Why” can be accusatory. “What” asks people to explain.

LDX3 London 2025 (Day 1)

2025-08-21T00:00:00+00:00

This June, for the fourth time, I was lucky enough to be able to head into London for the annual LeadDev conference, this time updated and renamed to LDX3. LeadDev is increasingly popular, and had outgrown the space at the Barbican, so this year was held at the Intercontinental at the O2. Previous years had separated the LeadDev, Staff+ and Leading Eng tracks, so if you had tickets to one, you couldn’t go and see talks on a different track. This year, it was one big conference, so apart from a separate Director+ area, anyone could see any of the talks.

As a space, the Intercontinental worked ok. The Barbican is a unique venue with a lot of character, so it’s difficult to replace that vibe. The Intercontinental was essentially a giant square hotel ballroom, surrounded by scrub land (although with a pretty cool view of Canary Wharf). The LeadDev team had done a good job jazzing up the space with lights and screens but the stages, even for the main Organisational Leadership track, were small compared to the Barbican, and I definitely missed the lovely comfortable Barbican raked seating, with perfect sightlines from wherever you chose to sit.

I’ve previously complained about the toilets at the Barbican, and those facilities were much better at the Intercontinental. Clean, functional and didn’t have to queue for long. However, food was a bit of a step back. With the different tracks breaking at slightly different times if your track finished later thanothers in the morning and afternoon breaks, you were likely to find all the food already gone. On the first day, there was no food left in the morning, and in the afternoon break I only found a sad-looking pot of chilli nuts. On day two, I happened to choose an early breaking track, so I managed to grab a tasty cake.

Lunch was better in that there was generally plenty of food, although I did spend about 10 minutes queuing near a pile of food because there were no plates available. Seating during lunch is always going to be difficult, and at the Barbican I usually ate while sitting on the stairs. The Intercontinental was no different. The desserts at lunch were similar to the breaks, and if you arrived late, you could well miss out.

The conference was split into three main stages: The largest Organisation Leadership stage, the medium-sized Technical Strategy stage, and the small Ways of Working stage. The LeadDev team underestimated the popularity of the Ways of Working stage, and on the first day quite a few people failed to get into the room as it was full. Credit to LeadDev though, by day two they had magiced up a hundred or so headphones and chairs so you could watch a live stream.

I appreciated the opportunity to pick and choose the talks that suited me best, so I did move around the stages over the two days. I planned my days to stay at the same stage until a break, and only then move. Unfortunately, a lot of people switched stages after every talk, which was very disruptive and occasionally disrespectful to the speakers who were on next. With the single-track conference at the Barbican, you didn’t have a choice but to sit there and watch whoever was on stage. If I were running the conference, next year I would advertise which talks were in each block, but not give specific timings for them to encourage people to not pick and choose quite so much.

As usual, there was a lot on apart from the main stages, including table talks, workshops, expert coaching and a networking event. I took part in a workshop, which I’ll write about in my round-up of day 2, and the networking mixer, but didn’t take advantage of the rest of the sessions. Away from the talks, the space was extremely busy and a bit overcrowded. It was not easy to move around. The sponsor hall, in particular, was always packed so I tried to avoid it unless I had no choice.

I think that’s enough rambling about the conference, the reason I went was the talks, so below is my summary of the talks I saw on day 1 of LDX3 London 2025…

The videos of the talks are available here if you sign up with a free LeadDev account.

Escaping The Rewrite Trap

Camille Fournier (Video)

Opening the conference is a tricky task as you have to set the tone for the rest of the talks. Camille is a LeadDev regular, and here was covering one of the topics that came up repeatedly over the two days - dealing with technical debt. Camille’s approach was that rewrites almost always fail, but reachitectures succeed.

The problem is that with any long-lived code base, it’s really hard to know what it actually does, so how can be you be sure you have reimplemented all the required features and logic? Hyrum’s law states that for a sufficiently large number of users, any observable behaviour will be depended on by someone - so if your rewrite changes anything, it will break for someone.

Camille made the point that if you have a large successful system, then the team supporting it is unlikely to be staffed for novelty. When you’re first building a system you have “pioneers”, who thrive on the hard work of building something from scratch, then “settlers” who start scaling it one. Once it’s successful, they are replaced by “town planners”, who are adept at maintaining and gradually growing the system. Asking your “town planners” to start from scratch is unlikely to be successful.

To be successful in rearchitecting you need to have, and clearly communicate a vision for the end goal. You need to have long-term goals, for example “features”, “efficiency”, “reliability” and “security”. Factor in the scale of the company, your rate of growth and migration costs - can you really afford this? What major things can you achieve in 12 months?

Once you have the plan, sell it to your team and sell it to your stakeholders. Take advantage of external deadlines, such as key technology going out of support soon.

Finally, Camille covered how to execute the plan. She first warned against bringing in a new hire to run the project. Although they may “have done this migration before”, they won’t know the team, the context or the system and will immediately be on the back foot. You should minimise disruption for your users, and work out how to ensure correctness. Can you test without breaking the rest of the company?

I felt this talk was a great conference opener. It had just the right balance of inspiration, a broad topic and actionable suggestions you can apply to your own role.

Rethinking Collaboration

Sara Dolganov (Video)

As with everywhere else, LLMs were a common topic at LDX3. In this talk Sara covered how to use LLMs to help in a variety of work contexts. She started with covering how LLMs can better structure your CV, and can help tailor a cover letter to the company you’re applying for.

Once you have joined a company she talked about how you can talk to an LLM instead of a rubber duck to help with getting unstuck while learning the new systems and technologies.

She also warned about where LLMs can make things harder - such as “skills atrophy” while vibe coding, making up facts in your CV and subverting the narrative if you ask it to rewrite a longer piece of text.

I’m pretty sceptical of AI, and this talk did little to change that.

How Monzo Tolerates Full Cloud Outages

Andrew Lawson and Daniel Chatfield (Video)

In this talk, Andrew and Daniel covered how Monzo avoids downtime, even when their primary system is completely unavailable. They began by talking about some recent large-scale outages. In the last two years nine UK banks have had a combined total of 803 hours of downtime due to IT outages. They also covered the UK National Air Traffic Control System (NATS) outage where a bad flight plan took down both their primary system, and because it was running the same code, so had the same bug, the secondary system too.

To avoid this problem, Monzo has what they call a “stand-in”. This is not a subset of their core system, but a completely independently implemented and hosted system that they can switch over to for anywhere between one and all of their users. It consists of 18 services (compared to 3000+ for their primary), and offers only the core features bank users rely on - checking balances, paying with a card, etc. It costs just 1% of their cloud spend to run.

They stay in sync by not modifying data in the secondary, but by keeping a list of changes that are then replayed on the primary when it is available. This avoids issues with the two systems becoming out of sync.

They convinced their executives to allow them to test on real customers, as this is the only way to be sure it works, outside of a major incident.

This was a really great talk. I thought they could have talked about how similar ideas have been used in aeroplanes and space flight, to give a bit of context, but regardless, this was a really interesting peek behind the curtain of a nationally critical piece of infrastructure.

Monolith-ifying Perfectly Good Microservices

Brian Scanlan (Video)

This talk followed a thread that was common to quite a few talks at LeadDev - that microservices are not a “clearly better, why would you do anything else?” solution, and instead, you have to weigh the tradeoffs with a monolith. In this talk, Brian described the journey Intercom went on with their Ruby On Rails monolith.

As is often the way they started with a monolith and scaled rapidly both in terms of team size and infrastructure. They started to suffer from stability issues, and flaky tests. They suspected that these issues were caused by the inability of Rails to scale, so they began to build new features in standalone applications and extracted billing from the monolith.

Even with this change in architecture they were still struggling with the monolith and team coordination was becoming a bigger issue. They focused on getting good at managing the monolith. They built tools to help with stability and worked on…

linting
dependency upgrades
test reliability and speed
code ownership
deployment
scaling MySQL
caching

After all that, the monolith was working much better, and coordination with the microservices was still a big problem, so they copied code from the services into their monolith.

Metrics, KPIs and Developer Experience: Rethinking Measurement For High Performing Teams

Oge Opara-Nadi (Video)

This was a great talk, and closely aligned to the challenges I am facing. They began by talking about how metrics are not strategy. Metrics are easy to measure, but hard to turn into actionable insights. They miss questions such as:

are we working on the right things?
are we reworking or getting blocked?
do developers feel trusted and supported?
is our current delivery pace sustainable?

They finished with a challenge. Audit one metric you rely on. What is it really telling you? And who is it serving? Is it helping your team improve? Is it aligned to your business goals? Is it reinforcing the culture you want to build?

Beyond “Try Harder”: Effective Strategies To Tackle Bugs

Andy Weir (Video)

Bugs kill trust, confidence and momentum. If you can build systems that make doing the right thing easier then the time you spend firefighting will go down, and the business will start trusting you.

This talk didn’t have much depth, and the suggestions are pretty obvious.

Human-first Leadership, AI-Powered

Sterling Chin (Video)

This was quite an interesting AI-based talk as it was a set of tools Sterling uses to make him a more efficient manager, and to allow him to be more present with his team, more present with his family and reduce people’s dependency on him.

He started the talk with a confession that he talked for five minutes into his phone and used AI to turn it into an outline for the talk. Before adopting AI he would wake up with 100s of unread emails and slack messages, which took 1-2 hours per day to deal with. His family felt the pressure because Sunday evening was Monday morning for his India-based team.

He felt that he was never present in meetings because there no break in between, and not enough time to context switch. The mental load caused by knowing what he was dropping was huge.

Sterling advocates for starting simple - make use of AI transcription and AI meeting summaries. If you had an executive assistant what would you delegate? An AI should be able to do that.

Use AI to free you up to ask real questions and make real connections rather than tracking sprint goals or other info.

Rather than your team asking you what was decided and waiting for a response, they can ask an AI trained on the meeting summaries.

Fast, Informed, Impactful: How To Master Decision-Making

Katrin Freihofner (Video)

In this talk Katrin started by describing a scenario where a team built some software, but used the “wrong” technology. Once an architect found out they had to rebuild it using the approved technology, which caused a big hit to morale. Katrin then described a fairly standard RFC (request for comments) process and talked about how you need clarity over who is involved in a discussion, and how you should use a template to drive the discussion forward to avoid repeatedly discussing the same topics.

Tracking Repository Health at Scale

Simon McManus (Video)

This talk was extremely relevant to things I am thinking about in my day job, so I was predisposed to loving it. Simon started with a discussion about what does healthy mean, and how do you know what constitutes your estate?

He talked about some “health principles”:

Secure customer data.
Easy to release, up to date and self-service.
Consistent across the estate.
Conformant - does it follow your rules?

Code health is about your code at rest, so it is not about DORA metrics or your KPIs. You can run static checks to determine the state of the code. For example, is the version of Node used an LTS version? Are the versions of the dependencies within two versions of the newest? Does it use any packages we’ve decided not to use?

Lastly, he talked about how to communicate the code health, and demonstrated a dashboard, and how they use that to show teams and stakeholders, and influence future plans.

Code Your Future

Germán Bencci (Video)

This was a short talk from the conference’s charity partner about how they give migrants software development training, access to support and recruitment help.

How To Delete Everything: The Clean-Slate Approach to Technical Strategy

Plum Ertz (Video)

Even if you were to going to get nothing from the content of this talk it would be worth watching for the classic Apple Mac slides on their own. Luckily, Plum is a LeadDev regular, and it shows. In this talk, she takes you through the decision process that many of us face - to debug broken code, or rewrite it?

Plum suggested taking an hour to consider your situation from a new perspective. Your current system has the advantage that it exists, but the new system doesn’t. Take on part of your architecture and discuss what would it mean to remove it? What inputs or outputs does it have? What processing does it do?

She then talks about how to approach a situation where your host is deprecating the version of Python you are using, and you have no choice but to act with a fixed deadline.

Rebuilding at Scale: A CTO’s Journey in a High-Growth Fintech

Lee Provoost (Video)

While I didn’t follow this talk or understand the point it was trying to make, Lee did introduce one concept that I like. Technical debt is well understood, but often it is caused by commercial debt. Commercial debt is caused by the hustle needed to get a business off the ground - such as bespoke deals that you didn’t originally anticipate.

Levelling Up: Transitioning Successfully into a Manager of Managers Role

Gisela Rossi (Video)

This talk pretty much does what it says in the title. It’s a step by step guide on how to prepare for, and start being, a manager of managers.

Gisela talks about two changes that happen when you move up - there are visible changes, such as the HR system updates and having more meetings and more access to confidential information. There are also silent shifts, such as the time horizon you need to work in expands, and there are changes in the bridges you need to build and maintain.

Perception

No-one is coming to brief you. You need to see the dots and connect them. You see enough of the detail, and see enough of the systems, to notice what others can’t. You need to use these to help your execs make good decisions.

Patience

Wins take longer at this level. 6-12 months is normal. The nature of the problems becomes harder - complexity, dependencies, messiness and urgency. You will end up with outcomes that no one is happy with, but that everyone can tolerate.

Presence

You are an ambassador for your teams. You have to learn to negotiate and advocate for them. Be a translator to help people outside your teams understand. Sometimes you won’t be in the room, but your thinking will be.

Frictionless Movement: How Internal Mobility Transforms Engineering Culture

Tom Murton (Video)

Small disclaimer with this talk - I was the technical adviser for LeadDev so it was great to see Tom had taken some of my suggestions on board.

In this talk Tom talked about how companies often make it easier to leave than to move internally. This creates unnecessary turmoil as people think they have to leave to grow.

Tom introduced a policy called “Transfer Windows” to help facilitate the moves. Some moves happened, but there was no mass exodus. The key thing was that people started to have conversations, and moving teams internally became an option.

Lost and Alone over the Pacific

Nick Means (Video)

Nick is the traditional speaker for the end of day one at LeadDev, and here he delivered one of his trademarked story talks. The topic for this year was a pilot who was lost over the Pacific, trying to find a tiny island in the dark, and how boats and other planes eventually helped him land safely. The moral at the end of the talk was that navigating the current AI landscape can feel like you’re lost and alone, but you’re not.

Water Monitoring

2025-02-13T00:00:00+00:00

In some of my previous posts, I talked about my journey towards monitoring every aspect of my home. For a long time now I’ve been measuring temperatures, electricity and gas usage, solar power and more. One area that has been on my target list from the start was our water usage - not least because a few years ago we had a leak that went undetected until it became a fairly significant problem. Recently while my plumber was doing our annual gas boiler service I asked him to fit a water meter which has allowed me to finally start tracking this data.

The key discovery that allowed me to do this was that you can get “pulsed” water meters (I used this model from B-Meters). They come preequipped with an inductive reader, and have a couple of wires you can connect to that receive a pulse for every litre of water used. Installing the meter was relatively straightforward for my plumber, despite the cramped space he had available.

I purchased a Raspberry Pi Pico W as it is easy to connect up the external devices, has WI-FI and runs a version of Python called MicroPython. I’m not the best at the physical side of builds, and my code is much neater than my wiring. I used a couple of alligator clips to link the meter wires to the Pico, and I put it all in a case to protect it.

The picozero library has a function specifically designed to count the number of pulses on a GPIO pin, so counting pulses is as simple as:

import picozero

PULSES = 0
def count_pulses():
    global PULSES
    PULSES += 1

switch = picozero.Switch(13, True, 0.1)
switch.when_activated = count_pulses

The key line is here is picozero.Switch(13, True, 0.1). The reason we’re using a Switch here is that the water meter has a switch that is magnetically closed each time the meter reads a litre of water. The parameters are saying that we’ve connected it to GPIO pin 13, that we want to use a pull-up switch, and that we want a debounce time of 0.1 seconds. You can use any general-purpose GPIO pin (a diagram can be found here), just make sure to connect one wire to a GND (ground) pin and update the number to reflect the pin you connected to. One problem that stumped me for longer than I would like to admit is that the pin identifiers on the board don’t match the GPIO pin identifiers. Make sure you map the physical pin you’re using to the correct GPIO id.

I’m not going to go into the difference between pull-up and pull-down switches. For a circuit as simple as this the built-in pull-up/pull-down resistors are sufficient, and I’m not aware of any practical difference between the two options. Please do your own research before wiring your circuit up to avoid frying your Pico!

Lastly, the debounce time needs to be set to a balance between the maximum expected pulse rate, and how long it takes for the switch to settle to a solid value. With a switch that is physically moving it can take a short amount of time before it stabilises, and if you read it too frequently you will see it flip between on and off causing you to count pulses that aren’t real. In my use case, a pulse every 0.1 seconds would work out to 36,000 litres of water every hour. The highest rate I’ve seen is just under 1000L/h, so this is well above the expected max, and seems to be long enough to avoid counting phantom pulses. You might need to experiment to find the sweet spot for your situation.

Once you have a counter variable, you need to expose the value to Prometheus through an HTTP end-point. Since we expect this to only be accessed by Prometheus, I decided to skip most of the HTTP spec and return just enough for the metric scraping to work.

wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect(WIFI_SSID, WIFI_PASSWORD)

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("", 80))
s.listen()

while True:
    conn, addr = s.accept()
    request = conn.recv(1024)

    conn.sendall(
f"""HTTP/1.1 200 OK
Content-Type: text/plain; charset=UTF-8; version=0.0.4
Access-Control-Allow-Origin: *
Connection: close

# HELP watermeter_count Total litres of water used.
# TYPE watermeter_count counter
watermeter_count {PULSES}
""")

The first block of code connects to the WIFI, then we open a socket to listen on port 80. In a loop we then accept an incoming connection, read but discard the body of the request (so whatever path and HTTP verb you use, you’ll get the metrics back), and send a preformatted response with just the count of the pulses substituted in.

You can find the code, plus a bit of extra code for error handling and reporting via Sentry.io on GitHub.

To finish, here is a screenshot of my Grafana dashboard showing several different ways of visualising the data. The cost is calculated by another application I have running, which uses the Prometheus metric and the known cost per litre (plus a standing charge) to work out how much our water usage has cost me.

LeadDev London 2024 (Day 2)

2024-08-15T00:00:00+00:00

This is the second part of my write up of the LeadDev London 2024 conference. To read my thoughts on the talks from day 1, and my overall impression of the conference please read my previous post. As with that post this is just the summary of my notes, so will give you a flavour of the talks. This should help you decide if you want to pay for a digital pass to access the videos, or if enough time has passed to look them up on YouTube or the LeadDev site.

In the afternoon I attended a workshop by a couple of Google engineers on DORA. I’ve watched the recording of the talks I missed, so I’ve included my thoughts on those below. My write-up of the workshop is at the end of this post.

And with that out of the way, on with the talks…

Germán Bencci and Seraphine Young

This was a talk about the Code Your Future charity. They do digital skills training for refugees and disadvantaged people. There was no leadership takeaway from this talk, but one fact that I was not aware of is that 1 in 78 people in the world is forcibly displaced. This is an awful lot higher than I would have guessed.

While the talk was fine, and the charity is certainly worthy they didn’t really have an “ask” at the end of the talk. I would have been ok with them asking for donations, but they didn’t do that, or talk about how you could get involved. I left the talk feeling glad that a charity like this exists, but not with a sense of how I could help.

Revolutionizing engineer growth: The tech-powered blueprint for careers clarity at ASOS

Gareth Waterhouse and Ed Collins

This talk was a case study about introducing a careers model as ASOS, the online clothing seller. It wasn’t clear what an engineer needed to do to get promoted, and they were getting low scores in employee satisfaction surveys. They also knew they had to support “squiggly” careers, where people move back and forth between different crafts as their career progresses.

They produced a set of conversation starters with versions for managers and engineers on how to approach career meetings, and what questions to ask. They also produced documentation that covered “What does good look like?”. It also included “How should <role> think and act?”, and “What should <role> do?”. They also produced a website where you could use sliders to measure yourself against different competencies, and then it produced a radar to highlight your strengths and weaknesses.

Making smart investments: A framework for maximizing your ROI in technical decisions

Katerina Iliakopoulou

Everyone makes technical decisions. Some are big decisions, some are small - but everyone makes them. Even if you want your team to make decisions, you have to give them guidance. Is how you define gain the same as your team or your manager? Bias affects how you see gains, and overlook costs so try to be impartial.

Katerina then presented a model for how to make technical decisions

Potential. Try to use metrics to identify the gains.
Risks. Identify constraints and uncertainties. Do we have the money and resources for this project?
Involvement. Who needs to support the project? Get their buy-in.
Skills. Do you or your team have the right skills?
Motivation. How excited are the team?

Risks + Involvement + Skills = Cost

Potential + Motivation = Gain

Iterate to greatness: Building high performance, AI-native engineering teams

Jared Palmer

This was a blatant pitch for Vercel, which in itself is not a bad thing, but it was so overblown that it just seemed like a puff-piece of AI hype. There was an interesting section in the middle of the talk about testing AI models and using data to improve and guide the training. One of the key points seemed to be that Vercel made it incredibly quick to deploy, on the order of seconds to deploy a new AI model. There was no talk about actually implementing functionality, it was just AI can do everything for you. Avoid.

Beyond the hype: Practical steps to establishing and scaling your data & ML team

Ellissa Verseput

In the age of AI dominating everything (see the previous talk!) this was a talk advocating for “traditional” machine learning and data analytics not to be forgotten. Ellissa works for a company that are trying to balance electrical grid supply and demand, which is important with an increasing share of green sources. They do this by turning large industrial appliances on and off.

The value proposition was established using a spreadsheet, and then they gradually evolved their data strategy by automating the many data requests Ellissa’s team were getting.

The Post Office Scandal: What we can learn from its process and human failures

Hywel Carver

This talk was a summary of the Horizon IT scandal, which caused more than 900 subpostmasters to be wrongly convicted of theft or fraud due to problems with the IT system. This was an excellent talk about the potential real-world dangers of an IT system. In 2000 it was decided that UK courts should assume that computers were working as intended, the same as they do with traffic lights and speed cameras. Anyone who knows how many bugs there are in computer software will find this terrifying…

The key message from this talk was that we should give our teams the right skills to do their jobs. We should listen to users and teams, and not bury our heads in the sand about potential problems. We should also trust people over systems, but verify to ensure our trust is well placed. And finally, we should be humble about our software.

The boss’s shoes don’t fit: And other surprises of leadership

Inna Weiner

This was a talk about the early stages of becoming a manager.

Speak less - you are likely not the most knowledgeable person in the room. Also, as a manager, your words will be amplified 10x. Listen more - if you avoid micromanagement you can become an absent manager, so be engaged and pay attention.
Speak more, but differently - explain the way you are thinking, and explain why something is important. Be curious. Diffuse the situation if necessary. You’re in other meetings, so be generous with the knowledge this gives you.
Connect the dots.
5:1 ratio of positive to negative feedback. (There’s a lot on this in other talks and books, but I’m not convinced it’s a good thing to follow.)
Be a good storyteller - accomplishments don’t speak for themselves.
Celebrate! Credit is infinite.

Uncertainty of change

Jitesh Gosai

This talk is worth watching for the slides alone. They zoom and slide over the screen, connecting one screen to the next in a very slick way. The talk started with some examples of people or companies who resisted change - the Luddites (who resisted textile machinery), Sears (who failed to adapt to online shopping), Blockbuster (who had the chance to buy Netflix and didn’t), and Nokia (who were sold to Microsoft and still failed to remain relevant).

Jitesh then moved on to talking about the psychology of change, and how people resist big change because it’s hard to predict, but small changes offer more certainty. They talked about our fight, flight, freeze and faun (i.e. submission) instincts, and how we can’t tell the difference between a change in a team’s ways of working and a bear.

They then moved on to the difference between complex and complicated. Complex systems are unknowable due to emergent behaviours, while complicated systems are difficult to follow and understand, but are ultimately knowable. They described how modern software is complex, not complicated.

Testing gives us certainty about how systems perform and should be focused on reducing uncertainty, not catching bugs.

Managing the marathon: Leading teams through lengthy migration projects

Lawrence Taylor

Migration projects are a staple of software development teams. They are coming for you whether you like it or not. To lead your teams through such a project you should focus on a story that will influence the outcome, and educate your stakeholders.

Highlight milestones as part of your storytelling:

Landmarks in the development
Capabilities that demonstrate new features
Integration milestones.

Become a better storyteller to engage, motivate and inspire your team.

The software bug all stars - and what we can learn from them

Christian Seifert

This was a quick run-through of some of the times software bugs have had a significant, even life-threatening impact. The cost of operational software failures in 2020 in the US was $1.5 trillion.

The Mars Climate Orbiter was lost after orbit insertion around Mars. The probe cost $327 million dollars, without taking into account the cost of the rocket. The probe was lost because it entered orbit 57 km above Mars, rather than 150km. Lockheed Martin were responsible for the orbiter, while NASA JPL looked after the ground systems. Lockheed used imperial units while NASA used metric (pounds of force vs Newtons) - they are a factor of 4.5 out.

The ground team realised there was a problem while doing testing as the probe was approaching Mars. They alerted management, but management ignored them as they didn’t follow the right channels. The issue was only found a few days before Mars arrival, so there was no time for the proper channels. The result was the probe was lost.

If you have a function like int getTotalAmount() how do you know what the type of the amount is? You need to try and avoid the need for implicit information, perhaps by using a MonetaryValue type.

The Therac-25 was a radiation treatment device which had a race condition that resulted in three people dying from a radiation overdose. The manufacturer claimed it couldn’t happen, even after it did. The race condition occurred when entering data too quickly. It took two years for the issue to occur because people had to learn and get quicker at using the system.

Be humble and don’t dismiss that it might not be you. Don’t just display error messages - display options and information so the user can make an informed decision about how to proceed.

Date handling is a huge source of bugs:

The Microsoft Zune stopped working on 31st December 2008 because 2008 was a leap year, and it only handled 365 days per year.
On the 1st of January 2012 Apple iOS alarm clocks didn’t work. This bug was caused by the fact that 1st January was in week 52 of 2011.
An F-22 Raptor fighter jet crashed when it flew over the date terminator because the software couldn’t handle the date going backwards.

How to set goals with people who don’t want to set goals

Alicia Collymore

This talk didn’t really match the title. It was Alicia’s process for working with her reports to set goals, but didn’t address how to get people who don’t want to set goals to engage with the process. Still, it sounded like a good process that some people might want to try out.

The first step is to do a brain dump of ideas:

Think big - do you want to be a CTO?
Think small
Include personal life.
Have they received any recent feedback?

Next, map each item to a timeline of short, medium and long-term. Then draw a line between items that depend on each other.

Finally, you need to shape the ideas and make them more specific so they fit your company’s goal framework (e.g. SMART, OKRs, etc).

Managing Engineering Teams in the Era of AI

Chris Class

While I’ve been pretty negative in my write-up of AI talks, this one was good as it was a clear case study of how they actually added AI to their product, and the issues they faced.

Chris recommended experimenting early so that when you want to add AI to your product the team is ready with the right skills and knowledge. The team was used to delivering features quickly, but AI needed a different pace. In particular, the unpredictability of AI meant a lot more time was needed to handle edge cases than was expected. They also learnt that minor changes in prompts made little difference to users, but took up a lot of development time.

If you’re not part of the solution, you’re part of the problem

Alessandro Canessa

This was a great talk, but difficult to take notes from. The idea is that being good is not enough, you have to drive change. Fixing other people’s mistakes is not a sustainable solution - you have to help people get better.

Mentorship + sponsorship

Lara Hogan

As is often the case with the final talks, this one was also hard to take notes from. When growing in our careers it can be hard to feel progress, mentors and sponsors can help both to grow your career, but also feel like you are growing.

Lara talked about the difference between coaching, sponsorship and mentorship.

Mentorship is great when you want help onboarding, or need unblocking.
Coaching is the best tool for internal growth.
Sponsorship helps you to find more opportunities.

To find a sponsor…

Do great work!
Find someone who knows your work.
Know how you want to grow.

Look for people who will help you grow!

The engineering leader’s playbook: A hands-on guide to DORA

Finn Toner and Rob Edwards

This was the 80-minute micro-workshop I attended on the Wednesday afternoon. The two Googlers, Finn and Rob, who were running the session did a good job talking about DORA and some of the findings as well as facilitating a discuss between the 30 or so people who were there.

The question being tackled was “How do you optimise value delivery from investment in tech and technologists?” They highlighted that DORA is not just the metrics most people know about, but it also looks at how the capabilities of a team predict their performance. Performance means both in terms of the reliability of their software delivery, but also how the well-being of their team members predicts organisational performance.

The key DORA metrics are:

Lead time for change
Deployment frequency
Change fail rate
Failed deployment recovery time

Teams that have a climate for learning have fast feedback cycles and make small changes.

They also talked about what change strategies work. Communities of practice and bottom-up or grassroots changes work well. Training centres, centres of excellence and big bang changes don’t work.

You can take the DORA QuickCheck to get an overview of where your team sits, and DORA Conversations is an interesting way to spark conversations on how to improve.

LeadDev London 2024 (Day 1)

2024-06-27T00:00:00+00:00

In what is fast becoming an annual tradition I recently switched on my Slack out-of-office, packed up a bag with plenty of room left for some conference swag, and got on the train to the Barbican for LeadDev London 2024 (see my review of 2022 day 1 and day 2, and 2023 day 1 and day 2). As with last year, this is a one-track conference focused on leadership and engineering management, with a separate one-track conference at the same time, StaffPlus, focused on Staff+ engineers. I only have a ticket to LeadDev, so I only got to see those talks.

As in previous years the event was held at the Barbican in central London. The main event space remains excellent, with good acoustics and sightlines for all 1000+ attendees. Last year I complained about the state of the toilets, and while the existing toilets are still there and haven’t been renovated there was a temporary set of additional toilets which were much better. The quality and speed of the food remains excellent, although I was a bit short-changed at lunch on the second day so needed to have a second dessert to ensure I had enough energy for the final set of talks (#FirstWorldProblems).

There was an expanded set of activities during all the breaks, but these were very popular and the terrible acoustics meant it was challenging to get close enough to even hear what was happening, let alone contribute. This fact was acknowledged by the host Meri Williams at the end of the conference, where the bombshell news that LeadDev will be leaving the Barbican after nine years was dropped. The venue for next year will be the InterContinental at the O2. Hopefully, this will solve the acoustic issues with the activities and they will have better toilets. Apparently, the move is to allow the conference to expand, so fingers crossed it keeps the lovely sense of community and doesn’t become a faceless megaconf (e.g. QCon). Time will tell, and even though the new location nearly doubles the commute time for me it’s highly likely I’ll be attending again.

One addition to the schedule compared to last year was a set of three “micro workshops”, which ran for 80 minutes at the same time as one block of talks. With three workshops, each running twice, there was plenty of opportunity to find a workshop that interested you and didn’t clash with talks you were keen on. As I attended the workshop on the second day I’ll cover it in the second part of this write-up.

In general, the quality of the talks was excellent, and even the first-time speakers did a great job of hiding their nerves and putting on a polished presentation despite speaking to over 1,000 people. I’m secretly glad that my talk was not among the 24 selected from 700+ proposals as I’d likely turn into an unintelligible pile of jelly if I had to walk on that stage, so kudos to everyone who presented. As usual, there was a good range of topics that are likely to be relevant to a wide range of leaders, and they mostly avoided sales pitches, with the notable exception of the Vercel talk on day 2 which I felt badly misjudged the room. It was also interesting that there was a lot of scepticism of Generative AI both on stage and in the audience, apart from a couple of talks (one of which was the Vercel talk). It’s good to see that is LeadDev is not getting carried away on the Gen-AI hype train.

What follows is a summary of my notes from each talk. I believe you can buy digital passes to catch up on the talks, or they are usually put onto YouTube after a few months.

Embracing engineering’s place at the forefront of business

Renee Hunt

This was an interesting talk by the CTO at Compare The Market about the overlap between technology and business. Half the executive team at CTM come from a tech background. Being a technology leader is morphing into business leadership as a tech person can learn about business, but you can’t give a business person 20 years of experience in technology.

She described how the number one priority for the business was a migration to GitLab. This is obviously a very technology-focused project, but all the executives (including non-technical execs, like the CFO) were on board with it being the top priority for the business because a good story had been told about why it was good for the business, and the future changes it would allow.

She closed with a message that people should prepare to lead, but even as a technologist, that role might not be directly in tech.

How do you deliver a feature on the biggest stage in the world?

Josh McNamee

This was a bit of an odd talk for LeadDev. It focused on the technology needed to display video at the Las Vegas Sphere, which is a giant dome with the world’s largest video screen. It’s certainly interesting technology, and it looks cool - but the talk was pitched halfway between a tech demo, and a talk about the organisation that helped it happen. As a result, it didn’t really succeed at either.

Engineering leadership in 2024 and beyond: Skating where the puck is going when the ice is melting

Lena Reinhard and Scott Carey

In this session the results of The LeadDev Engineering Leadership Report 2024 were discussed. The tech landscape has been complicated over the last few years, with waves of layoffs. Apparently, 7% of companies had both layoffs and hiring increases in the last year, and 71% of engineering leaders have seen the scope of their work increase. The main takeaway was that leaders should be “skating to where the puck is going, not where it has been”. The landscape is changing so fast, being left behind is a real concern.

Tech debt as innovation, reframing this forever problem as an opportunity

Bruce Wang

This was a great talk from a really engaging speaker. Tech debt is a natural byproduct of the software development lifecycle, but tech debt has a marketing problem. Tech debt doesn’t come from nowhere - systems that we are advocating for now, will become tech debt.

Bruce defined tech debt as the delta between a system’s current state and its ideal state. It is not business decisions you don’t agree with, or strategy misalignment. It’s not normal bugs and software errors, or even just bad code. It’s also not inherited code, systems or tools - just because you didn’t write it, doesn’t make it tech debt.

His recommendations for dealing with it include…

Be specific - what debt exactly are we talking about?
Make the debt visible, perhaps through metrics or reports that show where the debt is.
Be proactive - neglect is not a strategy.

Leaders change. Teams change. Business priorities change and technologies change.

Lies that people tell themselves about tech debt include:

Legacy code == tech debt (just because it’s old doesn’t automatically make it tech debt.)
Having debt zero as a goal.
Debt reduction work is only for junior folks, or as part of on-boarding.
There’s nothing we can do.

We should manage technical debt so we can say yes to the business more.

From overload to oasis: Creating more time for the cool stuff

Dominika Pietrzak

Cool stuff is relative, and changes over time. Share work that you consider not be cool to create space to do what you do consider cool, and because your boring is someone’s cool.

Managing across timezones; A retrospective

Matt Boyle

I found this talk decidedly “meh”. Timezones can be a real pain to manage across, but it’s not a particularly complicated issue.

The main points from this talk were:

Block out “golden hours” - when everyone is online. You won’t always use the slot, but you want it available in case you do need it.
Experiment with your processes.
Use async processes as much as possible - wikis, blogs, ADRs, messages in public channels.
Embrace timezones, and use them to reduce out of hours on call

Building for the new developer

Cat Hicks

This was a very academic talk, with a much more rigorous evidence base than usual. Unfortunately, I found it a bit unclear and unfocused, and I think it tried to cover too much given the timeslot for the talk.

When we join a new company we look for signals about what success means in that organisation. We use our biases and preconceptions about what a success requires in a field to decide who is brilliant - our biases and preconceptions can be wrong!

There was quite a lot about the AI Skill Threat and how it interactives with a thriving culture, and a contest culture in an organisation.

A learning culture drives up productivity.

Demystifying neurodiversity in tech with nostalgic video games

Parul Singh

This talk had amazing slides with a classic video game vibe. Unfortunately, the video game theme didn’t seem to apply to anything other than the slides, which felt like a bit of a missed opportunity.

They said that 53% of people technology are neurodivergent, but only 3% are disclosed. To help people work better small accommodations make the world of difference. And don’t be afraid to make suggestions - neurotypical people can make good suggestions too.

Try to take advantage of people with a different point of view, and a different way of thinking - dissent opens the mind. Be careful though, because unresolved conflict leads to a disengaged team and lower productivity.

Only 1 in 4 people is diagnosed as a child, so they are often learning what works best for them. Some people produce a “user guide” or a “working with me manual” which describes how they like to work and helps to avoid conflict.

How to use technology radars to make transparent tech decisions

Andra Blaj

Most people question a technology choice when they join a company.

To create tech radar:

Get leadership buy-in.
Document all technology choices, including retroactively.
Use human-centered design to make the radar easier to use.
Build a culture of documenting technology choices.

You should use the tech radar to question established technology choices.

Building a tech radar is easy, but building a culture of documenting decisions is hard. It takes time to embed that in a company culture.

Delivery metrics - the good, the bad and the utterly ridiculous

Jennifer Mackown

This talk was about measuring the performance of software teams, although it didn’t include as many examples as I had expected based on the title. Jennifer talked about the importance of distinguishing between measuring “What are we building?” vs “How are we building it?”. One key measure she mentioned was the “goal ratio”. This is based on the fact that we know the sprint goal, and we know the work in the sprint. The ratio of work that is goal related against total work in the sprint is the “goal ratio”.

Measure for change

Laura Tacho

Measuring and improving are two different things. If we want to improve we must change the way we measure. We should measure for change, not for control. To drive change it is important to involve your team, tie data to decisions, and always follow up on actions.

Quantitative data is easy to measure, but usually, it’s not clear how to improve. Qualitative data is much harder to measure, but makes it much clearer how to improve.

Software is made by humans, for humans, and disregarding the feedback from the humans (developers) making the software only gives you half the picture. Developer productivity is complex - quantitative data is rigid, and you can only measure what you think to measure.

Developers know where the problems are, so just ask them. You can correlate sentiment with data from your build tool.

This was a talk from a conference sponsor (DX) so it was clearly a talk that was justifying their product, but it was not that obvious, and Laura focused on the problem rather than how their tool helps to solve it.

Data science demystified

Grishma Jena

This was a strange talk. As the title implies, it was a basic overview of what Data Science is and how a Data Science team works, so the talk did what it set out to achieve. I’m just not sure why, and I’m not sure who in the audience wouldn’t know what Data Science is.

You are here: The story of the Barbican

Nickolas Means

As was the case with the previous two LeadDev London’s that I’ve attended the first day was closed by Nickolas Means, who is a brilliant storyteller. In previous years he has talked about Fukushima, the Boeing 737 Max and Three Mile Island. In a change of pace this year he didn’t talk about a disaster, but about the history of the Barbican. He started by dropping the bombshell that this would be the final year of LeadDev at the Barbican (although the new venue wasn’t revealed until the end of day two), and then worked his way from the end of World War II through the design, planning and building process to the present day. I will always recommend Nickolas’ talks, although in this case it’s probably more interesting if you have a connection to the Barbican. The general management theme he drew at the end was a bit tacked on, and not as immediately relevant as in his previous disaster talks.

What Do You Want From A Team Meeting? A Decision Or A Team?

2024-03-21T00:00:00+00:00

Most teams will have a team meeting on a regular schedule. Once a week, every two weeks or once a month the team gets together to discuss the day’s big issues. This can be a difficult meeting because it doesn’t have a defined agenda, and is open-ended.

Other meetings either drive a project forward, solve a specific problem or implement a specific process. They might have a varying agenda, but the overall goal of the meeting is (hopefully) clear, and there is a point when the meetings will stop. Barring the team disbanding, this is not true for team meetings.

A common pattern, and one I am certainly guilty of, is asking people what issues they would like to discuss, and perhaps sharing a few updates from around the business. When you get to the end of the agenda you just say “Thanks everyone” and go back to your desks.

The problem with this is that you are encouraging people to just bring existing problems to the table, and then only if you manage to get the culture to be non-threatening enough that they feel comfortable with sharing their issues with the group.

What if you were able to get the team to work together on topics before they become a problem? If you could build relationships that foster collaboration outside the regular meeting, and allow people to be proactive rather than reactive.

In the book Time To Think by Nancy Kline she talks about creating a “Thinking Environment”. Now, this book has quite a few problems (to put it quite mildly) and I’m not a fan of it, but the core idea is sound - by giving people the space, time and attention to think clearly they will reward you with insights that may lead to a step change in their, and your team’s, performance.

Team meetings between individual contributors and those between a team of leaders have a different focus. When it is ICs collaborating the team should be working on the same domain. While they might have focus on different areas, or have different skillsets, the challenges they face and the goals they are working towards will be similar. For a team of leaders this is not necessarily true, and the differences between the areas represented in the meeting can mean there is little overlap in the challenges being faced.

In both cases though you should aim to mould your team members into a team. Where more disparate people are involved this will be harder, but can be achieved by reiterating why you are part of the same team, and what the common objectives are. Either way, you want to turn the group of people in your meeting into a team.

So how do you do that? First, open your meeting with a positive “icebreaker”. Just asking people to share one positive thing that happened to them in the last week, and letting everyone speak in turn will set the mood of the meeting, and show that everyone’s contributions are welcomed.

Once everyone has had a turn to speak, move on to the agenda. At this point, you need to decide what you want out of the meeting this week. It could be that there is an urgent problem that needs a decision; in that case, you should steer the group towards making it. However, often the conversation is more about deciding the strategy the team wants to take while tackling a project or problem, which will shape the future work.

While discussing strategy getting as many views as possible is key. No idea should be unheard, and while you do eventually need to agree as a team on the desired approach, at the beginning of the discussion the more inputs there are the better. Once someone has shared something, allow everyone to comment and to discuss it. Whether you do this by explicitly going round each member of the team, or more subtly asking those who haven’t yet spoken what they think doesn’t matter, just make sure everyone contributes.

After you have contributions from everyone, and people have asked clarifying questions of each other, either yourself as the moderator or the person who raised the topic should summarise the discussion, and list the action points. These should be assigned and follow-up dates agreed upon.

One aspect of building a team is that people have to feel that they are jointly contributing to achieving success. If other people in the team have completed a project or solved a problem, then others will (hopefully) be happy for them, but unless people can see how they achieved that outcome together, it won’t help build a team. As part of your meeting bring back topics that were previously discussed and mention how the discussion contributed to success. Perhaps the action points were completed, the discussion brought to light an alternative approach or highlighted potential problems that were avoided. Whatever the reason for the success, if it is tied back to the work of everyone as a team, team connectedness will improve.

As a leader, it’s easy to feel that problems that are most acute in or only affect, your teams should be dealt with by you. Building a leadership team that shares problems and collaborates on solutions benefits everyone, and can only happen if people share. Encourage people to consider the future of their team, and ask for feedback from their peers on things that they don’t consider problems. If they are regularly sharing and discussing their teams with their peers then they will make sure they are constantly challenged to be the best leader they can be, and when a genuine problem does arise in their team they will be well placed to resolve it.

As the moderator of your team meeting, make sure you know when to guide the team to a decision, when to encourage discussion and problem-sharing, and when to create space for everyone to be heard. The team meeting is your key opportunity to set the tone that will influence how your team interact with each other for the rest of the week, so don’t waste it.

How do you ensure your team meetings are productive and help you build team spirit? Let me know what you’ve tried, and how it went, in the comments below.

Balancing Standardisation With Innovation

2024-02-01T00:00:00+00:00

In my earlier post Sustainable Software Development I talked about how to build software that can be maintained and improved over the long term, and how to run teams that can work without burning out. In this post, I want to expand on one area of this - how to maintain innovation while using standards to reduce diversity in your code base.

Snow-flake applications, which have unique characteristics compared to others in your estate, take a significantly outsized amount of time to maintain. This extra complexity requires specialist knowledge in your developers, and the time taken to context switching between applications is increased. You also don’t benefit from the economies of scale, where you can apply improvements across many of your applications simultaneously.

Historically, sharing code across applications would be done through shared libraries, but the modern software development landscape is more complex and many companies are moving to Platform-as-a-Service models (PaaS). This is where the infrastructure used to run an application is more than just a server with a CPU, some memory and a disk. Instead, it is a whole ecosystem of APIs, developer tools and a platform for running code. Whether internally developed, or purchased from an external company, PaaS systems are chosen because they abstract developers away from much of the drudgery that previously they would have had to work through, but this also constrains them to only using services provided by the platform.

PaaS is not the only form of standardisation though, companies will usually have policies (whether implicit or explicit) that they only use certain languages, frameworks, third-party libraries or design patterns to build their applications. However standardisation is enforced, the end result is the same - developers trade off increased speed of delivery and reduced cognitive load against reduced freedom to write code as they wish.

What happens if you want to do something that doesn’t fit with your PaaS platform? Or perhaps your company has standardised on Java but you would like to write some code in Kotlin, or maybe Python? Maybe a new build tool has been released that you think will offer quicker builds or simpler configuration, speeding up your cycle time. Whatever the reason, changing the status quo is hard, but feeling like you are forced to stick with what is currently used is devastating for a developer’s morale.

As a leader, you need to balance the benefits of standards against the threat of stagnation. While standards give you benefits with current development, you also need a defined policy to change the standard, and a process to update existing code to match the new rules. If you don’t then your teams may be missing out on improved ways of working as the technological landscape evolves.

To allow for innovation you need to create time for developers to experiment, with the expectation that the code written will often not be deployed into production. Whether or not an experiment is successful is not important - what matters is that you learn one way or the other. This innovation could take the form of 10% time, or just giving developers flexibility to add experiments onto the team roadmap with low friction. While hackathons are a good way to build experimental new apps, they aren’t good at changing standards. Firstly they usually come around too infrequently, and with the goal of building something that you can demo they don’t lend themselves well to more technical or process experiments.

As well as providing time for developers to experiment, you also need to supply motivation. If developers see ideas for improvements being shot down without proper consideration, if they don’t understand why they were rejected, or worst of all if suggestions languish with no decision either way then people will not choose to put themselves forward to experiment. Instead, they will feel like they have to suffer with the current processes, even though they believe that it can be improved. This is terminal for morale, and productivity.

Once you have created a standard, and provided space to propose changes to it, the next step is to document the process for proposing those changes, how they will be judged (and by whom), and how they will applied. How this will work depends on the size and structure of your organisation. You might have an individual architect or head of engineering who would make sense as a decision-maker, even better would be a group of developers who can act as an architectural council. There should be a list of criteria that will be applied, so people know what to address in their proposal. Some examples could include the effect on running costs, the learning curve for other developers, or the effect on the speed of feature implementation (perhaps by reducing the number of lines of code that need to be written).

Applying a change in standards to your existing code base can be painful. It’s important to approach it actively though, otherwise code that follows the old standard just becomes technical debt. You need to identify areas where the legacy method is used, and create a project or projects to update them. This won’t happen overnight, but scheduling this alongside a team’s other work is needed to avoid the old code languishing until it is forgotten about.

How do you handle standardisation in your teams? Have you found a way to encourage innovation while maintaining a coherent code base? Please let me know in the comments below!

Pimoroni's Interstate 75W

2024-01-11T00:00:00+00:00

I’ve previously written about my attempts to get metrics about my house into Prometheus and to visualise them with Grafana. This project has gone well, and I can measure everything I want to apart from water usage - please let me know in the comments if you have a suggestion on how to do that! The one thing that’s missing is a “glanceable” display. My Grafana dashboards work well when I’m at my computer, but they’re not so great when cooking in the kitchen.

To solve this I purchased an Interstate 75W from Pimoroni, and a 64x64 2.5mm pitch LED matrix panel. This is a Raspberry Pi Pico W, with some additional software and hardware to make driving LED matrix panels easy. The Pico W contains an RP2040 processor, 264kB RAM and 2MB of flash memory, a USB port and wifi/bluetooth support.

Assembling the panel is easy as the board slots into a space on the panel easily, and connects with a couple of provided wires. Just don’t forget to connect the power cable as well as the data cable (as I did), as without it it powers up but the colours look very strange and you’ll be confused for a while.

Getting started with the board is straightforward as Pimoroni provide an excellent guide, and a custom version of MicroPython with some baked-in extra libraries to make drawing on the display straightforward. It was only a matter of a few minutes after unboxing before I had something appearing on the panel.

The additional libraries have a lot of useful functionality, like drawing lines and shapes, images, text, etc. However, I quickly discovered that the development process was not quite as slick as I would like. There’s no way to run your code on your PC - it will only run on the physical hardware. Given I would like to mount the panel in my kitchen, having to plug it in every time I wanted to work on a change was going to be impractical.

Partly to solve that inconvenience, and partly because I don’t like to make things easy on myself, I built my own library, i75. This builds on top of Pimoroni’s MicroPython libraries, but only uses the ability to set individual pixels. It also abstracts the Interstate 75 away, so when running on a PC it mimics the panel using PyGame.

This approach has several problems, in particular, it runs on Python3 which MicroPython attempts to replicate, but the standard library is very different. The Pimoroni’s build of MicroPython only emulates single precision floating point numbers, which caused me some trouble when drawing an analogue clock. And of course, the speed of a PC is about a zillion times quicker than the RP2040, and has essentially no memory limit. Ideally, in future, I’d like to switch to the PC MicroPython build, but for now, the simplicity of using PyGame wins out. I’ve developed the glanceable display without using the physical hardware, so as long as you keep these limitations in mind it works ok.

As I write this I have not yet implemented all the displays I wanted to, but it does display the time, some bouncing balls (mostly as a test of speed), arrival and departure times from my local train station and the album art from whatever’s playing on my Kitchen Sonos.

To further improve the speed of development, and to work around the memory and processor limitations the display has a frontend and backend component. The backend is standard Python 3 running in a Docker container on my Synology NAS, and it does all the communication with downstream systems, and only returns minimal JSON for the display to show the required information. MicroPython includes a cut-down version of the requests library, which lets you make HTTP requests and parse the returned JSON.

One interesting challenge was handling Sonos album art. It was pretty obvious that the frontend wouldn’t be able to cope with loading the image directly, but even after retrieving and resizing it memory issues were a problem. The backend loads the album art and track information using SoCo, and then resizes the album cover down to 64x64 using Pillow. I had assumed this would be small enough to send to the frontend easily, as it’s only 12kb, but when I stuck it in the JSON response I would get out-of-memory errors. Processing the JSON allocates quite a bit of memory, but it’s still surprising it fails as there is 120kb available. Instead, I moved the loading of the art to a separate request and passed it as a raw byte stream. This avoids the processing needed to load the JSON, and will only use up 12kb of memory. The Pimoroni libraries support PNG and JPEG, but as mentioned earlier I don’t like making things easy for myself, so I’m trying not to use more features than I have to.

Text handling has been another area I’ve struggled with. With just a 64x64 display getting legible text is difficult. You don’t want too much writing on a “glanceable” display, but for displaying data like song and artists names, it is important. After trying a lot of fonts I decided on CG pixel 3x5, which allows for about 10 lines of text. It only supports capital letters, but for all characters, you need a much denser display to be able to distinguish them clearly.

Unfortunately while dealing with network resources errors are inevitable. By default console output on the Interstate 75W is lost, when not connected to a computer. To work around this I implemented the code below, which attempts to catch all exceptions, post the traceback to my backend server (which logs it) and then restarts. This works well as I can track any errors that occur, and it keeps running rather than needing to be power cycled. I have still experienced some issues with hangs, but I think those are to do with network calls missing timeouts. In future, I might try to add support for Sentry, but for now this keeps things running in event of bugs.

def main_safe() -> None:
    while True:
        try:
            main()
        except KeyboardInterrupt:
            break
        except Exception as e:
            s = StringIO()
            sys.print_exception(e, s)
            s.seek(0)
            log_error(s.read())
            time.sleep_ms(1000)
        except:  # noqa
            log_error("Unknown exception...")
            time.sleep_ms(1000)

def log_error(error: str) -> None:
    try:
        r = urequests.post(f"http://{BACKEND}:6001/error", data=error)
    finally:
        r.close()

I’ve had the panel running for several months so far, and it’s been very reliable. The network connection is stable, and there have been no problems that were not due to my code. The display is very bright and clear, although it would be nice to come up with a box or frame to hide the wires and make the whole package a bit neater. In general, I highly recommend the Interstate 75W if you’re looking for a display for a home project.

As I continue to develop the library and the frontend I will post about any interesting challenges I come across, in between my management-focused posts. Have you done any projects involving an LED display? Let me know in the comments below!

Sustainable Software Development

2023-12-07T00:00:00+00:00

When you’re in the zone it can be hard to think about the future of the code you’re writing, but the time you’re spending writing it is just a fraction of its lifespan. Code may be written once, but it will be edited a few times, perhaps copied once or twice, and likely executed many hundred, if not thousands or millions of times. You as a software developer will (hopefully) have a long and productive career in software development, as will the team you work with. And finally, you also do your work on planet Earth (hi to any readers on the ISS), and without it you’d really struggle.

When creating or modifying an object or system that we intend to last for a long time, it is crucial to consider sustainability. This includes factors such as the sustainability of the source code, your or your team’s ability to work on it for an extended period, and the software’s impact on the environment. All of these aspects are significant and should be carefully considered.

If you’re working in a scrappy start-up, fighting for survival, then the long-term maintainability of your code will not be important to you. In most cases though, code can be expected to last for a significant time, and the cost of maintaining it will significantly outweigh the initial cost of writing it.

Code that is not maintained will accumulate problems. Libraries it relies on will have new releases, as will the operating system it runs on. These will provide new features, fix bugs and solve security vulnerabilities. Software that has not been touched for a while requires significantly more time to make simple changes, compared to software which is regularly altered.

The DevOps movement helps significantly with maintainability, as any manual process ages very quickly. Even with excellent documentation, it isn’t easy to reliably follow a process you’re not familiar with. Automation, particularly the build, test and deployment steps, are well worth the time spent on building them. If you can combine a tool like Renovate with a high-quality test suite, and a continuous deployment pipeline, you can focus on other things, safe in the knowledge that your application is being kept up to date with the latest security fixes.

Beyond automation, how you write code will affect how sustainable and easy to maintain your code base is. This doesn’t mean you should add comments everywhere, indeed if you need comments then your code could probably be clearer. Not that there aren’t times when comments to explain why a section of code is written as it is are needed, but they should be rare.

When writing your code, think about how it will be read, and how it will be maintained. This starts with the language you use, as well as the libraries, frameworks and design patterns you choose. No language, library or framework is perfect for all situations, but there are bad choices. The two things to consider when making a choice are how much knowledge of the technology there is in your company (or how much time you are willing to invest in training), and what the community of the project is like.

While you might think Kotlin (to pick a random example) is the perfect choice for your new application, if no other apps in your company use Kotlin, and knowledge amongst your colleagues is low then it will be hard to maintain in the future. Similarly, with frameworks and libraries, the ability for other developers to jump between applications and focus on the logic and not the surrounding implementing details can hardly be overestimated. Languages and libraries don’t by themselves deliver value. They might enable you to deliver value quicker compared to other choices, but only if you can unlock that advantage, and maintain it long-term.

The other aspect is the community around the technology, and where you are comfortable sitting on the technology adoption lifecycle. It’s certainly a good idea to keep abreast of new developments and experiment with them by building things you’re happy to throw away once you’ve learned about them. But betting your company on a brand new library, that’s only existed for a month and has one contributor, has a high risk of you depending on code that’s been abandoned and either needing to migrate away from it, or commit to supporting it yourself. At the other end of the lifecycle, if you wait too long to adopt, or stay with a dependency for too long, then it may move out of support and be abandoned too - exposing you to unpatched security vulnerabilities, and conflicts when other dependencies drop support for something you are using.

Most software development projects last many months or years. For people working in an Agile fashion, there might not even be a specific end goal, just an endlessly evolving set of requirements and software that is growing to meet them. Burnout is a significant issue in software development and as a leader, it is your job to recognise the signs and work to prevent it. If your team burns out, productivity and quality will collapse, and the costs will be much greater than if you had worked more sustainably.

Common reasons for burnout are poor working conditions, pressure and unrealistic deadlines, and a culture that doesn’t prioritise learning, improvement or balance in the demands on team members. This is a huge topic, far too big for one blog post, but sacrificing a bit of speed now will result in happier developers and a much more sustainable development process.

One of the most demoralising things about being a developer is working on software that is hard to change, but you don’t have the permission, or time, to make the needed improvements. By dedicating an appropriate amount of time to tech debt reduction you not only end up with more sustainable software but a more sustainable culture of development too. As a leader, you might need to defend this time, but by looking at security vulnerabilities in dependencies, or the risk that you will no longer be able to get support should you need it, it is possible to build a strong business case for this work.

The most common thing your code will do is run. Whether it’s executing a thousand times a second, or once a week when someone runs a report, it’ll spend more time running than you spent writing it. Computers use energy, need materials to build, and cost your company money to run. If you can halve the time it takes a process to run you’ll give your customers a better experience, save your company money, and reduce the impact your code has on the planet.

A centralised database clearly has a significantly smaller environmental impact than a blockchain-based ledger - even if you take into account techniques like Ethereum’s Proof of Stake. Which one is better depends on what your priorities are, and what tradeoffs you are willing to make. They are tradeoffs though, and they demand consideration during your design process.

Not all tradeoffs are quite as extreme as the database vs blockchain example. You might be working on a report which does some algorithmic processing of data. If you’re writing in Python it’ll likely take longer to run than if you replace that section with highly optimised C code. Again, there are no right answers, but the Python code will likely be significantly easier for your team to maintain in the future, so perhaps that is worth the extra costs incurred by the longer run time.

It’s easy to focus on the joy of building a complex system, one that will be useful to your customers and make money for your company. But an often overlooked customer is you in the future, or those who come after you and have to look after the system you are now building. When you are choosing the technology you will use and the systems architecture you’ll implement, make sure you consider how easy it will be to maintain, how your team will work without the risk of burnout, and the impact your design choices will have on the planet.

Have you chosen one technology over another because of the ease of maintainability? Have you changed architecture because of environmental concerns? Let me know in the comments below.

Small Celebrations, Big Motivation

2023-11-02T00:00:00+00:00

Software development can sometimes feel like an endless treadmill. Push that commit, review that code, investigate and solve that bug. Rinse and repeat, day after day.

You probably have some high-level business and technical aims that you’re working towards. These might be releasing a large new feature, or perhaps migrating to a new infrastructure or a new software stack. These long-term projects might take a year or more to complete, and can often feel never-ending.

If you’re working on a mature code base it can be difficult to see the progress that you’ve made. One of your jobs as a leader is to help the team step back and see how far they’ve come, and one way to maintain motivation in a team is to celebrate achievements and show the wider business the progress that has been made.

It can be hard to know how to celebrate, and how often. Too frequently and you’ll be celebrating things that are too small and simple, and it will seem forced. If you wait for the large projects to finish then you could be waiting for a very long time, and it will feel like you never celebrate anything.

I’m not saying never celebrate the small stuff, and never celebrate the big projects. If someone has completed a particularly gnarly ticket, then it’s worth shouting about. Take the team out for a meal if you’ve successfully launched a big new feature. However, it’s the in-between celebrations that are key to maintaining motivation in a team.

There are two audiences for a celebration - internal and external, and both can have motivational benefits.

By celebrating internally you can help people to feel recognised by their peers, to step back and see the progress they’ve made and help people to realise that those large, long-term goals, which may seem unattainable, are actually achievable.

Internal celebrations don’t need to be big or fancy. Even a short Slack post calling out some good work to a wider audience will be appreciated. If you have monthly or quarterly town halls, for a small group of teams then these can also be a good venue to call out a successful piece of work.

External celebrations might take the form of a town hall to a wide audience, an email newsletter or a global Slack post. The aim will be to raise awareness of the work your teams have been doing, to a group of people who might not be aware of the details.

As well as raising awareness of your team’s work, a side effect is to show people that their work is valued by the wider business and to show that you as a leader are acting as a cheerleader for your teams. People want to know that leaders are advocating for them, which most of the time happens privately - in budget or roadmap meetings, so you shouldn’t turn down the opportunity to highlight their good work in public.

Getting the business to truly value your work can be a double-edged sword. Having the CEO, or other high-level manager discuss a project, or even announce a goal related to it, is unmatched in the ability to help people feel like their work is important to the company, and to focus minds on exactly what you are trying to achieve. However, this level of attention can feel like the eye of Sauron and is not always welcome - especially if the work is particularly challenging or aspects of it are unknown.

I believe the benefits of having a senior manager engage publically with a project your team is working on far outweigh the potential issues, so make sure you’re using your meetings with them to advocate and sell your teams. Encourage them to use their platform to talk about your teams, and try to get goals at the organisation level that unambiguously relate to projects that you’re working on.

Have you got any suggestions about how to celebrate small wins? How do you raise awareness of the teams you work with? Please let me know in the comments below.

Leadership Stand-Ups

2023-09-28T00:00:00+00:00

Standups are standard practice among developers, but when you move into the realm of leadership they stop. If you’re a team leader then you will probably join your team’s stand-up meeting, but often you only contribute a brief update, if anything at all. Mostly you just listen and perhaps facilitate discussion on other people’s updates.

In Five Dysfunctions of a Team by Patrick Lencioni (affiliate link included) the CEO in the story asks her executives to consider that group as their primary team. I’ve been asked to do the same too, but really, if you only meet once a week or perhaps even less frequently, how can you do that? If a group is your primary team then you should know (at a high level) what everyone is working on, and what their main problems are. A leadership team where you only discuss what people decide to bring to the table will not function as well as a group that can benefit from collective wisdom being brought to bear on problems.

When you start a leadership position your calendar suddenly fills up with meetings, so the idea of adding a daily meeting might seem crazy, and it probably is. However leadership is a different role to a developer, and you can use different processes to achieve the same goals.

As a developer, simplifying greatly, you might work on one ticket, then move on to the next, all working towards completing a small number of projects. Leadership positions usually involve much longer running pieces of work, with many more running in parallel. I typically might touch on four or five of my current agenda items, progressing each one a little bit each time. There’s not much use in giving a daily update, a weekly cadence shows a better view of my progress.

As a leadership team you probably already have a weekly meeting, so why not spend the first 15 minutes of that doing a stand-up? You could, but it’s probably quite an expensive 15 minutes, and a rushed, or not well-thought-out update does not deliver any value.

With remote working now being much more common the tools for asynchronous communication are far more widely available and used. Whether it’s an email, a Slack post or something else a weekly post that’s shared with others will have benefits for yourself, and your leadership team.

For the last three and a bit years, I’ve been posting a weekly update on Slack, based on the concepts of the 5-15 updates - a message that takes you 15 minutes to write, and five minutes to read. Much like with blogging (where I’m far less good at keeping to my schedule) the habit of writing every week, on a Friday afternoon has been very beneficial.

The act of writing each week and organising my thoughts on what I’ve achieved greatly increases my satisfaction that I’ve made a positive contribution, or sometimes forces me to realise I need to do better next week! I divide my updates into two sections, what I’ve done, and what I hope to do next week. By publically committing to doing certain things next week I hold myself to account for them, and allow others to contribute if it’s a topic they’re interested in.

By both documenting what I’ve been working on, and what I’m going to do others in my department are able to see what is important to me, and either help or suggest things that I should be focusing on instead.

Finally, while I’ve not had to write a brag document in the sense that Julia Evans describes. I have found an archive of what I’ve been working on useful on occasions. Looking back through the updates it’s clear that I’m not the same person I was when I started, and that’s a great self-motivational tool.

One problem with being a manager is that you occasionally get involved in secret work. This might be an HR process, or perhaps some company announcement that is not yours to announce. Hopefully, these are few and far between, and for the HR work you should never make that public. Once something you have been working on has been published then you should include an update that describes your contribution, so the historical record is complete.

I see great value in writing a weekly update for yourself, and if as a manager you can model good behaviour then others will follow suit. As a team of leaders, more communication, more collaboration and more alignment on what you’re working on can only be a good thing. Let me know what you think about weekly updates in the comments below! Have you tried writing them? Have you commented on other people’s updates?

LeadDev London 2023 (Day 2)

2023-09-21T00:00:00+00:00

This is a follow-up to my first post about day 1 of the LeadDev London 2023 conference. That post covers my notes on talks from day 1, as well as my thoughts on the overall conference.

Here are my notes on talks from day 2…

Building bridges: The art of crafting seamless partnerships between engineering, product, and design

James Stanier, Winter Wei and Janet Balneaves

This was an interesting style compared to the other talks at LeadDev. Instead of one or two people delivering a prepared talk, here we had three people (one from each discipline of engineering, product and design) having a discussion. It worked pretty well, and they had clearly prepared and made some useful and interesting points. As it wasn’t a prepared presentation it perhaps wasn’t as impactful as the other talks, but it was definitely a nice change of pace.

Riding the rollercoaster of emotions

Gabriel Michels

Some emotional events that can happen at work are:

Breaking production
Team member unexpectedly quits
Users don’t like a new feature
Difficult co-worker

Start with yourself - handle your own emotions first. Recognise negative thoughts, keep calm and don’t react emotionally.

Platform engineering is all about product

Gal Bashan

Dev and Ops -> Dev and DevOps -> Dev and SRE - rebranded teams, same problems.

The goal of platform teams is the enablement of developers.

Have a problem-first mindset - ensure you are solving problems your developers are having. Developers know what problems they are having - ask them!

Even with platform teams, agile is still valuable. Deliver incremental value and get fast feedback.

Cultural post mortems: an approach to learning and recovering when your people systems fail

Winna Bridgewater

Complex adaptive systems operate at the edge of chaos. We work in organisation complex systems, with…

a shared goal
varied interactions
feedback loops
emergent behaviour from teams operating as a system.

Strategies for succeeding as a underrepresented engineering leader

Rafia Qutab Kilian

Leadership is a learned skill set.

Assess a company like you would a neighbourhood.

Expand your network - sponsorship is more valuable than mentorship.

Be strategic!

Feature flags unleashed

Roger Gros

Flags should be part of a company culture, but they do create risks…

breaking current behaviour
operational complexity
tech debt
high initial investment

Build a data-driven on-call workflow for your team with atomic habits

Bianca Costache

Create a sense of shared accountability and empowerment.

Look at the mean number of interrupted nights.

To improve the on-call experience tackle:

Siloed knowledge
Poor documentation
Boring technical adversity (where all fixes lead to large problems)

Make sure alerting is connected to service level targets. You don’t want to be alerted for issues that are not causing a problem.

To improve avoid the status quo bias - “Would you like a shovel? But I’m already using a spoon!”

Atomic habits:

Make it obvious
Make it attractive
Make it easy
Make it satisfying

Create space for reviewing on call.

Keeping your team health after a layoff

Leandro César Silva

Prepare for the event:

Plan what work will be dropped
Prepare the communication
Draw scenarios and team structures
Take diversity into account
Talk to direct leaders

Through the layoff:

Talk to impacted people immediately
Talk to the company as soon as possible
Talk to teams as soon as you can after you’ve talked to the company

Give people time to say goodbye. Don’t make another big change for a while.

Moving forward there will be…

Fear - reinforce the company’s next steps, and ensure people maintain a good work-life balance.
Guilt - reinforce why the layoffs happened.

To help retain those who remain:

Recognise good work
Implement shared projects
Prioritise opportunities internally
Give people some time off
Some people will need mentorship - be closer to them.

Parents who code: How to welcome your developers back after parental leave

Sinéad Cummings

Sleep bad = code bad - and as a new parent, you are almost guaranteed to be sleeping bad.

People who were working while someone was on leave will have seen changes be introduced slowly. The person who was on leave will see them introduced overnight, when they return.

Try to limit the knowledge gap created by long leave through documentation. Keep a record of changes for them to review when they return.

It is understandable to be resistant to change when you were not involved in the discussion. Try to show the options you’ve considered.

Assume nothing about what matters to the returnee.

Use KIT (Keeping In Touch) days to spread information over the period of their leave.

Assign them a buddy who will advocate for them while they are on leave, and support them when they return. This is optional, people might not want a buddy.

Creating inclusive career ladders

Sally Lait

A progression framework is a compass, not a map. If you have a checklist you’ll end up with a specific view of what a good engineer looks like.

Focus on outcomes, not prescribing how. If someone doesn’t want to give talks, they could find ways to share their knowledge with others.

Time is relative. You might say people need to demonstrate behaviours for six months, but for people on parental leave or who are part-time, this might not work. People should demonstrate behaviour over multiple quarters, but it is important to give people context about why time is important.

Building for the underserved, solving for all

Serah Njambi Kiburu

W3C four principles of accessibility:

Perceivable
Operable
Understandable
Robust - must work with a variety of assistive technologies, browsers and devices.

Driving positive change through performance improvement plans

Cristina Yenyxe Gonzalez Garcia

Performance Improvement Plans are one of the most challenging processes to navigate. Everyone understands the importance, but no one is excited about the prospect. They usually happen too late and are rushed. The aim should be to set the employee up for success.

Stage 1: Before

Give initial feedback.
Try to avoid the PIP, but equally don’t wait for multiple occurrences of the behaviour.
Actively work towards the employee’s development.

Stage 2: Initiation

Ensure they understand the full meaning.
Be clear about the good and bad outcomes.
Choose measurable outcomes to decide the result of the PIP.

Stage 3: Duration

Celebrate improvements.
Don’t let them think things are going well went they aren’t.
- Give feedback
- Don’t be too nice
Keep a written record.

Stage 4: Wrap Up

If it didn’t work out…

Have a final review of unsatisfied criteria.
Clarify next steps

It went well…

Celebrate success.
Follow up to ensure are continuing to go well.

Exit plans and how to talk about them

David Kiger

The average tenure of someone is three years, but we assume people will stay forever.

Having conversations about exit plans allows people to move ahead with confidence. It is okay to talk about exit plans! Talking about will help you avoid surprises.

Managers are better than most people at leaving because it happens to them so often.

What if people think those who aren’t sufficiently committed will be punished? Make it clear that people have options in the company, and support people who decide to leave.

Avoid getting to the point where you have to make a last-minute save, as they rarely work for long.

It is much cheaper and easier to engage with and retain current staff than to hire.

You need a strong foundation of trust to have these conversations.

Orchestrating thousands of bots from the cloud

James Donkin

As I work for Ocado I can’t really comment on this talk. It’s worth watching if you want to see some of our snazzy technology though!

The awful agony of the app store: When software delivery goes wrong

Clare Sudbery

As with last year, the conference was closed by a talk from Clare Sudbery. While I enjoyed that talk, I didn’t connect with this one and didn’t understand the point of it. It was about her attempts to build a mobile game, and the nonsensical rules and interactions with Apple’s App Store. The only interesting idea I took away was the concept of a walking skeleton - a Hello World app that you can deploy, and then iterate on.

LeadDev London 2023 (Day 1)

2023-08-17T00:00:00+00:00

Recently I was lucky enough to attend Lead Dev London for a second time (see my review of 2022 day 1 and day 2). As with last year, this is a one-track conference focused on leadership and engineering management. Unlike last year LeadDev was also running a separate one-track conference at the same time, StaffPlus, focused on Staff+ engineers. As I didn’t have a ticket to that, I didn’t attend any of those talks but it was great to see LeadDev expanding and I’ll be considering StaffPlus next year.

The event took place at the same location, the Barbican in central London. The auditorium is a lovely space, with excellent sight lines from everywhere and good acoustics. The spaces outside are pretty tired looking, and in particular, the toilets were terrible. For a 1000+ person conference, this took the shine off attending. The food provided was good, and the logistics of the lunch rush were handled well. This year the networking mixer was held downstairs in the sponsor’s hall, whereas last year it was upstairs in the breakout space. I felt the sponsor stalls got in the way rather, and prevented people from mixing. I think the addition of something to help people mix would help break people out of their company groups - whether it’s a kind of speed dating system or something a bit more low-key, just handing people a beer and leaving them to it doesn’t work.

On stage, things were generally slick, with Meri Williams MCing again. The conference suffered from a few technical issues, with clickers repeatedly not working, a few microphone issues and some issues with videos playing without sound. Some issues are understandable, but it was a bit disappointing that they hadn’t been ironed out by day two.

Compared to last year the conference has added several activities in the gaps between conference sessions. These included live coaching, table talks and AMAs with LeadDev staff. I only tried attending a table talk, and it was too noisy to properly engage in, which was a real shame. To help navigate all the activities two new “experience hosts” were added (Jessie Belle and Amber Shand, who were presenters last year). They added a great feeling of energy to the proceedings, but having three people on stage at the end of each session of talks was a bit slow, and most people just want to get up and stretch their legs. Hopefully, LeadDev will find a better way to promote the extra activities. Even just having Meri at the start and Jessie and/or Amber and the end of each session would have made things run more smoothly.

Those a minor niggles though, and I hope to be attending again next year. Anyway, onto my summary of some of the talks. LeadDev will put the videos online at some point, but as I write this they are only available to people with tickets.

Managing at the threshold: Examining our principles in a moment of change

David Yee

In 2002 Google removed engineering managers and switched to a much flatter org structure. This lasted for a few months, and a few years later in 2009 Google published Project Oxygen which describes what they had learned about engineering management.

For years it was accepted that the most senior engineer in the room would be your manager, but engineering management is a skill in its own right.

After COVID-19 there was a period of change known as “The Great Resignation”, which was people taking control of their careers.

A Liminal Space is a space between states. It is space where we change. If you are in a Liminal Space nobody is going to tell you want to do. If you don’t have a process you can trust, you have principles you can follow. Stay true to those.

Notice
Think
Persuade
Proceed

Compassionate on-call

Lisa Karlin Curtis

Flexibility

You shouldn’t have to stop to go on call.

Autonomy

Anyone on call has to be able to fix their own pain.

Support

Anyone on call must have support they can use.

The foundation of sustainable on-call.

Red 2.0: Transforming a game company

Colin Walder

This was a talk about the process of changing a games company after the disastrous launch of Cyberpunk 2077, which was riddled with bugs and poorly received, even after the team had worked a significant amount of crunch time on it.

Unfortunately, this was a pretty bad talk. There was some talk of how the teams were restructured, but not in enough detail to be useful to others. There was some talk about how they adopted agile processes, but this was not a talk about agile, so lacked details - and how many people other than games companies are not already using agile?

This talk might have been interesting if it was after the release of the Cyberpunk expansions that is coming later this year, so they could compare the development process and reception to both, but since it’s not been released yet there is not much to compare yet.

What do you mean there’s no onboarding plan for engineering managers?

Daniel Korn

Starting a new position is hard, and starting as an engineering manager is doubly hard because as a manager you will have people immediately looking to you for guidance, inspiration and support, despite being new to the company.

The key factors for a good onboarding plan are creating a sense of belonging, and delivering a sense of accomplishment. It should prepare people for their new role, and connect them to the culture.

Things that should not be part of the plan are making technical contributions, and making big changes.

New employees should own their own onboarding, so they are in charge of the pace and content.

Code is poetry

Niranjan Uma Shankar

Maintaining a clean code base can have a huge impact on team performance. It will help you to:

find bugs early
add features quicker

Try to encourage strategic programming over tactical programming. Schedule time for spike sessions before you tackle a big project, because the first idea might not be the best. Recognise strategic programming with rewards.

All code should look like one person wrote it, so normalise nit-picking - but be kind!

Why onboarding to a company’s legacy codebase sucks, and how to make it work for your team

Shanea Leven

When onboarding you need to learn a number of things…

About the code/system:

how does it fit together?
what are the domain objects?

About working with the code:

deployment procedures
how to test

But you need to remember that people are human! Memory retention is an issue, and it’s hard to learn new things.

Treat every code review as a mini onboarding to that part of the code. Start small, and expand the scope strategically. Choose tasks that will build confidence in the codebase - tasks that require learning a small number of new things.

Tech debt reduction can be a good first task. Upgrades need little knowledge.

Learn about the frameworks used, as well as the implementation.

Making the move to manager: Common pitfalls for new engineering leaders

Jacqueline Pan and Marlena Lui

It’s ok to say “I’d don’t know” - be yourself!

Know where your time is best spent, and at what level you need to know the details.

“I’m happy where I am” - Supporting team members that aren’t seeking progression

Ryan MacGillivray

There is a myth of constant promotion.

The practice of an art isn’t to make a living, it’s to make your soul grow

– Kurt Vonnegut

Recognise cycles of ambition in your reports, and try to support them when they are ambitious and when they aren’t. Try to create a sense of development - grow in their role, not just move up.

Don’t deprioritise these people.

How to drive pace in your team

Alicia Collymore

Pace is value delivered at speed.

Set some defaults:

what are you willing to compromise on?
what are you not willing to risk?

Give lots of room for autonomy, and be transparent by default.

Create some hype! Get people excited about the value they will bring. Keep projects very short, and keep them simple - use tried and tested technology. Aim to do the hard bits first, and remember that work done is more important than work started.

Be laser-focused on value. We’re not here for fun, we’re here to make money!

Aim to beat deadlines - where can you get the best value for the time left over?

Pain is expected, and that’s ok - but try to give people on-call a break!

There are risks with being this focused on value:

too much pressure
poor code quality (so aim to make active tradeoffs)
unhealthy competition
- not everyone is fast in the same way

How we support making architectural decisions

Olena Sovyn

This was a description of their current process for making architectural decisions. They have seven members of Technical Design Architecture Advisors. They have different domain knowledge, and they rotate members regularly. They aim to be advisors, not gatekeepers.

This article from Stack Overflow was recommended as it describes how to write a technical design document, and gives a good structure to start with.

Once a change has been proposed they spend 3-4 days gathering further details, and 3-4 days discussing them in the group before making a decision. The possible outcomes are:

Sponsored - they approve this change.
Needs update - the proposal needs more work.
Not recommended - they aren’t gatekeepers, so can’t say no, but they suggest not to do this change.
Review not needed - this is an area they don’t cover.
Escalated - for some reason, this needs to be looked at by senior management.

Where we’re going wrong with developer productivity

Cat Hicks

Productivity is the stuff that is hard.
Productivity is never stopping.
Productivity is lonely.
- we need to work together!
Brittle productivity.
- productive if nothing goes wrong.

Resilliant productivity
- when something goes wrong, it delivers.

Thriving means:

Agency
Learning Culture
Motivation & Self Efficacy
Support & Belonging

Good managers blame processes not people.

Good productivity is adaptive and sustainable.

Engineering A More Equitable Hiring Process

Jason Wodicka

Equity = treated differently to enable the same outcome.

Tolerance = You are allowed to be here.

Inclusion = We want you to be here.

Avoid pedigree bias - where they went to school or worked.

Decouple assessments - clarity of speech and explanation vs just communication.

How to effectively “Spike” a complex technical project

Aditya Bansal

First, understand the problem, then understand the technical landscape.

Don’t…

write clean code
solve for everything
take all the time in the world

After the spike…

what’s new?
what’s being modified?
what do we need to discuss?

“Those with knowledge don’t predict. Those who predict don’t have knowledge.”

The 9.1 magnitude meltdown at Fukushima

Nickolas Means

Nickolas gave the closing talk to day one of LeadDev last year, and this talk was equally excellent. It’s impossible to take notes for, but do watch it if you can find it online.

LeadDev London 2022 Day 2

2022-07-29T00:00:00+00:00

Recently I was lucky enough to go to LeadDev London. You can read about my general thoughts, and the first day of talks, here. This post covers the second day.

The following is a summary of my notes from each talk. Any mistakes, errors or things that don’t make sense are entirely my fault. Hopefully, this will provide a flavour of each talk and inspire you to watch the ones that interest you the most to get the full story.

Why we are writing a monolith, not a microservice

Supriya Srivatsa

This talk was particularly relevant to me as we have a lot of monolith applications, and we constantly talk about the best way to split them into microservices… but maybe we don’t need to?

A key part of good architecture is abstraction, but it doesn’t need to be over a network.

At Atlassian they are building a modular monolith. Boundaries are hard to define, but you can use domain driven modules to split the code base.

A Commune in the Ivory Tower? - A new approach to architecture decisions

Andrew Harmel-Law

The typical way of making architectural decisions is that a team requests a decision. The architects go and have a think, then the decision is received by the team, and they try to implement it. The problem is that this doesn’t scale and can significantly delay the team.

In this talk, Andrew Harmel-Law proposes a decentralised way of making decisions - The Advice Process.

Anyone can make a decision, but before doing so they must seek advice. There are two parts to this process - a lightweight ADR (architectural decision record) and the Architecture Advice Forum.

The ADR is a document that describes a decision that was made (or is in the process of being made). It should contain:

The title
Context - you should put real effort in so people can understand why you’re making this decision.
Consequences of the decision
Advice - who gave what and when.

The Architecture Advice Forum is a weekly meeting to discuss in-progress ADRs and for people to give and receive advice.

How To Fail

“Bad” decisions - how do people deal with decisions they disagree with? Give advice, but let the team do what they want.
Old guard is the new guard - if you don’t get new people writing ADRs
Off-the-grid decisions

The last of these is quite serious as people are not following the process. You should ask them to write an ADR and find out why they didn’t in the first place. Did they not realise it was a decision?

Sorry… you go ahead. The art of making space and claiming space in meetings

Jemma Bolland

Only 35% of people say they can contribute in meetings when they have a contribution to make.

Some people think to talk and others talk to think. Don’t let your meeting be dominated by the latter category of people.

Make sure everyone knows how you would like them to contribute, and before you attend a meeting, if you’re not confident, find out what areas you can best contribute to.

Leave space before moving on to the next point - ask “is there anything I haven’t considered?”

Using incidents to level up your teams

Lisa Karlin Curtis

Incidents…

broaden your horizons.
teach you to fail gracefully.
teach you to make systems easier to debug.
are a chance to spend time with, and learn from, the best.

To help people learn make incidents more accessible. You can lower the bar so you declare an incident more frequently. This will help you to practice for the big incidents when they do come.

Encourage everyone to participate by announcing incidents and sharing a public slack channel.

While resolving the incident show your working - share as much as you can, including what you tried that didn’t work so people can follow along after the incident.

Be the catalyst in a junior engineer’s career

Amber Shand and Jessie Auguste

There are four stages to a person’s career in a company:

Attraction
Onboarding
Development
Retention

The Tech Talent Charter is a non-profit organisation aiming to improve inequality, diversity and inclusion in the UK tech sector.

Mentors vs Buddies vs Sponsors

Mentors should avoid spoon-feeding people - teach people how to teach themselves.

Buddies should cooperate with them. Be a safe person for them to discuss things with.

Sponsors should aim to identify and promote high performers.

Provide opportunities for people to add value beyond adding code.

Compassionate Refactoring

Clare Sudbery

Refactoring should make things…

easier to understand.
so they can fit into our heads.
cheaper to modify.

Move in tiny steps, and don’t refactor code that isn’t testable.

Why do people not refactor?

External pressure.
Unrealistic deadlines.
Impatience.
Seeking perfection.

This leads to guilt… shame… and despair.

This makes you feel like you’re not as good as the book you read, or the speaker you saw.

It makes you want to hide bad code.

It makes you try a big bang refactor, which doesn’t work.

We’re in a knowledge industry and yet we find it hard to ask permission to spend time to think and analyse.

People in leadership feel like they need to give the impression of everything they do being great. This makes junior engineers feel bad.

We need to allow time because we are fallible.

Sculptors start a raw material and remove bits. Writers and coders need to create the raw material so we can start editing - the first draft.

Something that is slightly improved is better than something that has been destroyed to try and make it perfect.

Learn how to make the tiniest improvement that keeps the tests green.

Seek the joy of coding and of doing a good job. People who enjoy their work are much more effective.

Skiller Whale Coding Competition

During the conference Skiller Whale were running a coding competition where the top three entries would win an Oculus Quest 2. The challenge was twofold. Firstly you needed to write an SQL query processing some customer data. The second part was to automate the submission so you could submit it quickly enough.

The first part was pretty fiddly to get right. There was a customer table and an orders table. The problem was to calculate the average rank of customers over a period, where customers are ranked daily based on the amount spent, including on days when they didn’t spend anything. Ranking customers is easy enough using the rank function. The challenge comes from including customers on days they didn’t order anything. This requires a full outer join between a table of dates and the customers and then left joining onto the orders table.

Each time you submitted an entry you were given a different date range and target rank to return. The second part of the challenge is to automate the submission so you can submit in less than one second. Luckily for me, only two people completed this part, leaving the third spot open to the person with the most efficient query. I managed to sneak into third place by replacing the date table, which initially I was doing this by running a distinct over the dates in the order table (requiring a full scan of the table), with the generate_series function which is much more efficent.

I’ll write a review of the Quest 2 when I’ve played with it a bit, thanks Skiller Whale!

I’ve included the SQL I submitted below, just in case anyone is interested.

with sales_by_day as
    (select dates.date, customers.id as customer_id, name,
            rank () over
                (partition by dates.date
                    order by coalesce(sum(amount), 0) desc)
                as pos
        from customers
             full outer join
                (select @start_date
                        + (n || ' day')::interval as date
                    from generate_series(0, 30) n) as dates
                on 1=1
             left join orders on
                orders.date >= @start_date
                and orders.date <= @end_date
                and orders.date=dates.date
                and orders.customer_id=customers.id
        group by dates.date, customers.id, name)
select name, avg_pos from (
    select name, avg(pos) as avg_pos,
           rank () over (order by avg(pos) asc) as avg_rank
        from sales_by_day group by customer_id, name
) as avg_rank
    where avg_rank=@rank limit 1

LeadDev London 2022 (Day 1)

2022-06-17T00:00:00+00:00

Last week I was lucky enough to spend a couple of days at the LeadDev London 2022 conference. This is a series of conferences aimed at senior engineers and engineering managers, although for obvious reasons it’s been a few years since it was last held.

The conference itself was held in the Barbican Hall, which in general is a lovely venue. The seats in the hall were comfortable, and spending two days sitting there was not as unpleasant as I had feared. The Barbican itself was built in the early 1980s, and a few areas, in particular, looked like they needed some refurbishment - particularly the downstairs men’s toilets.

The food was very nice, particularly the Banoffee pie from lunch on day two. The size of portions was generally good, although you did need to stock up from the salad bars at lunch to get a decent plate full. Queues were also not too much of a problem, other than the first ten minutes or so of lunch.

Enough about the practicalities though, what about the content? No one goes to a conference for the food, but instead for what you can learn and take back to your job. LeadDev is a single-track conference, so you have no choice over what to see, but fortunately, most of the content was general enough to be useful to everyone. Both days were hosted by Meri Williams who did a great job keeping things moving swiftly, and setting the scene for each talk.

The one talk I did skip was about scaling a mobile app deployment process because I don’t work in mobile. There was another talk with a brilliant title about driving and defining software quality, which turned out to just be a story of someone’s time in game development QA, and I struggled to find any useful takeaways. These scheduling missteps were rare though, so now let’s talk about the talks I enjoyed the most.

As I write this you can buy on-demand access to all the talks from the LeadDev site. I don’t know if they plan to make the talks available for free at some point, but previous conferences are available, so that suggests they will be.

The following is a summary of my notes from each talk on the first day. Any mistakes, errors or things that don’t make sense are entirely my fault. Hopefully, this will provide a flavour of each talk and inspire you to watch the ones that interest you the most to get the full story.

Navigating The Chaos of Scaling

Vitor Reis

This talk was about how to tell what a high-performing team look likes, but also about how to make the trade-offs to make high performance sustainable.

What does chaos look like? It can look like managing 20 people and still being on the on-call rota (depressingly this applies to me!) or doing five or more interviews a week because there aren’t enough people available to do interviews.

Try to recognise when you’re doing poor work, or are wrong, by being self-aware. Also, ask your manager what they think it would great looks like in your position, and compare it to your current performance.

Your aim is to be replaceable. If there’s a process you run can you automate it? Can you delegate it?

Be efficient with your time, and take the time to reflect on how you’ve spent your time each week, Try to do better each week!

It’s tempting to add people to a team but this can have negative consequences. People start to wonder if they’re being replaced. They worry that the new person might not do their job well, or that they might do it better than them! Adding people will make things go slower, at least for a while.

As a leader, you should be writing all the time. Writing is thinking, and it helps others to follow your thought process.

Keeping Your Code Base Fun At Scale

Raul Chedrese

This was a short talk looking at the question of how we ensure every developer has a fun experience? Code bases naturally become less fun to work on as they get bigger.

Don’t assume you know the answer - try to understand the developer experience, then form a vision and iterate on it. Understanding can be as simple as sending a survey with the question “What is preventing you from having a fun experience?”

Focus on building strong tools. Pick one thing and focus on that, rather than trying to solve too many things at once.

CSS: Cascading Support Systems

Phil Bennett

Empathy in senior leadership is a superpower. It increases creativity and productivity. When you are several levels up it’s easy to burn out worrying about the problems of all the people who report to you.

A technique that Phil has borrowed from therapy is Solutions Focused Brief Theraphy. This works in a similar way to how we work in product and engineering teams, so is familiar to people. To follow it you ask the question “Imagine the problem you’re facing has gone away. What do you notice?” Work with them to define the steps to get there, and help them to execute those steps.

As a senior leader, you should ask people to support each other, and cascade this to the people who report to your reports. You can scale empathy!

What Dashboards Don’t Tell You

Laura Tacho

As someone who is passionate about dashboards and trying to be data-driven in management, this was my favourite talk of the conference.

Some metrics can be perceived differently. Laura used the example of peaks and troughs in a tickets completed metric. Without additional context different people can interpret the data differently. By overlaying the number of incidents the metrics become more valuable.

People’s self-perception matters more than we think, and we should not discount qualitative metrics.

How do you move from metrics being perceived as surveillance rather than empowerment?

Vanity metrics are metrics without meaning. We can add tension to metrics to make them more valuable. Balance speed or activity against quality, reliability or satisfaction.

Laura recommended three sources that I have not yet checked out, but I intend to.

She also recommended GetDX.com as a tool to help you measure developer experience. Unfortunately, it has a truly terrible website, with no information about the tool, or any hint of pricing, just a “request a demo” button. I really liked the sound of this tool, but it’s going to be hard for me to justify investing time in it.

Building The Perfect Asynchronous Meeting

Alexandra Sunderland

How is an asynchronous meeting different from a Slack thread or an email? It’s still scheduled like a meeting, but instead you just put deadlines into people’s calendars about when they should have responded to the meeting. It requires all the usual good practices for meetings, but they’re even more important. You should share timelines for the meeting, expectations about what people should do, logistics of how to contribute (Google Docs, Miro, etc), and what you expect from people.

Meetings don’t need to be 100% async, but the preparation can be done before a shorter synchronous catch-up. For a retro, you could fill in the retro board ahead of time, and then have a meeting to discuss the board.

Meetings can be asynchronous for information pushing or brainstorming, but relationship building should be synchronous.

The benefits are that you get more time for thought, as the context is more accessible (although more effort is required to set the agenda and in keeping notes). The downsides are that it requires more coordination, and you don’t get real-time answers or fast-paced discussions.

This was a really interesting idea, and I intend to try it out for the next retrospective I run.

How to build trust as a new manager in a fully remote team

Sadhana Gopal

Being remote means you must be more intentional about building relationships. You are only represented by your active voice, so don’t be passive.

It’s easy to misinterpret when remote - assume positive intent.

As a manager try to design the wider environment, then design the team environment, and finally design yourself to model the behaviours you want to see.

Use asynchronous meetings for productivity, but synchronous meetings for engagement.

Don’t be always on - acknowledge FOMO, and take time to switch off.

People Building: Career planning for your direct reports

Daniel Burke

On a practical level, this was a really great talk. Daniel Burke was a really engaging, enthusiastic speaker who really owned the large stage in a way that most other speakers didn’t manage.

The great resignation is not about people quitting, it’s about people growing. This talk was focused on Daniel’s method of career planning. He asks people three questions.

Position - where are you?
Plan - where do you want to go?
Progress - how will you get there?

Aligning on performance with your manager is essential to moving forward on career building. You can only be successful if you’re on the page as your manager about what success looks like.

Don’t wait for the role to open up to begin preparing for it.

As a manager, it’s important not to project your motivation onto others. Although it’s a dirty word, money is a valid motivation for going for a promotion!

It’s not about the next level, it’s about the next step.

Taking the 737 to the MAX!

Nickolas Means

This was a really great talk about the two Boeing 737MAX plane crashes, the failures that lead to the plane design that allowed them to happen, and precisely why they crashed. The talk was really hard to take notes from, and isn’t directly relevant to many people, but is well worth a watch if you have time.

Don't Call It Tech Debt

2022-02-03T00:00:00+00:00

A common refrain from developers is “We have too much tech debt! We need to tackle it to speed up!” It’s hard to argue with the sentiment - badly written code or outdated software does take considerably more effort to maintain than code that follows current best practices. However, saying we have too much tech debt is not useful, because it’s not specific enough. You can’t create a vague “tackle tech debt” project and expect to get sponsorship from the business for the work.

If you’re lucky enough to have a portion of your time available to be used for engineering directed projects then you’re still unlikely to be successful with a vague tech debt project. Getting agreement for what tech debt actually is is nigh-on impossible. Everyone has their own pet peeve that they will want to tackle as part of that project.

My solution to this problem is to avoid using the phrase tech debt. Sometimes tech debt is clear - it might be a database that is outside of its support lifecycle or a library that is many versions behind the current release. More common is that when people say “tech debt” they mean things like code that is not as testable as they would like, or which follows some patterns that they declare to be an anti-pattern. While there is broad agreement about what constitutes good code, the more detailed you get the more it becomes about personal opinions.

The key to a good technical project is the same as with a good business-driven project - you need clear aims, a well-defined definition of done and a coherent set of steps to get you there. Let’s look at the three types of technical projects outlined above in turn.

The simplest case is upgrading an outdated database or library. Getting alignment on the goals of the project can be achieved by creating a policy around the supported version. Perhaps you want to always be running the n-1 major release version, or treat anything that has a new version released more than two weeks ago as outdated. Whatever rule you decide on it’s relatively easy to apply the rule across your codebase (particularly if you use something like Renovate) and come up with a well-defined project to address that debt.

Let’s now consider the case where people say that code is as not as easy to test as it should be. This is much harder to define a rule for, compared to outdated versions. What you can do though is define coding standards to enforce testability. Perhaps you always want to use dependency injection to make it easy to mock dependencies out. Maybe instead you’re worried about your tests being too high-level and therefore too fragile. Your coding standard should define acceptable patterns which can then be compared to your code to provide a list of targets for refactoring. Providing you obtained enough buy-in and agreement when defining the standards there should be little argument about what code to tackle. This is likely to highlight a large number of targets, so prioritising based on a measure of test flakiness is probably worthwhile.

The final case I want to consider is the more general case where people suggest that your code contains “anti-patterns” that need to be refactored. As with “untestable” code “anti-patterns” is not a well-defined word. To create a project to tackle this type of technical debt first try to agree on what is meant by the term “anti-pattern”. It’s easier to start from the positive side of this debate - set up a group of developers to discuss and agree on what libraries, patterns and other code style guidelines you want to encourage. The group should write this up and get agreement from the whole team that they buy into the recommendations.

Depending on the scale of your “anti-pattern” challenges you might want to tackle only a small portion of your issues initially, to avoid doing too much work upfront before you start to see the benefit. The working group could tackle just one or two issues initially, and later expand the guidance.

Once the guidelines have been published compare them to your existing codebase and keep track of where your code deviates from the recommendations. Depending on the scale of your challenges you might want to analyze your code at a higher or lower level. Using automated tools to find and track deviations will give your more specific targets to fix.

Tech debt is a negative phrase that is hard to get agreement on. By turning the discussion on its head to focus on the positive concept of code quality you can encourage more productive conversations, get more alignment in your team, and ultimately run successful projects to tackle issues in your codebase.

Have you had any good or bad experiences trying to tackle tech debt in your codebase? Let me know how you got on in the comments below!

The Power Of Dashboards As A Management Tool

2022-01-13T00:00:00+00:00

In November 2021 I gave a virtual talk as part of the IT Non-Stop 2021 Conference. Here is a recording of that talk.

Office Based Osmosis

2021-12-03T00:00:00+00:00

Recently my collegue Chris Sheldon posted an article titled “Under Pressure”. It’s excellent so please take a few minutes to have a read of it.

Ok, are you back? Great. Chris talks in part about feeling like he misses absorbing knowledge through osmosis in an office.

Osmosis, in case you’ve forgotten your biology lessons, is the spontaneous passage of something through a semipermeable membrane. In this case, we’re talking about knowledge entering your head without you needing to consciously do anything about it.

Working from home can indeed be an isolating time. In my experience, short term productivity and intra-team relationships are (in general) not affected by working from home. The biggest impact is on inter-team relationships and a lot of that is down to the osmotic acquisition of knowledge that Chris was talking about.

Back in the distant past (e.g. 2019) when you were sat near another team you would overhear discussions about what they were working on. Whether consciously or not, this would lead to you instinctively knowing when there was overlap, or when there were opportunities for collaboration. It would also lead to a feeling of working for a larger group than just your team, and this would result in a closer connection to those in other teams.

If you miss out on this knowledge then it can result in duplicated work or conflicting solutions to the same problem.

So, what can you do to improve things?

If you aren’t going to acquire knowledge spontaneously, then you need to be more intentional about both obtaining and sharing knowledge. The key point here is that you must be more deliberate about acquiring knowledge, but it’s also important to be deliberate about sharing knowledge so others can benefit too.

In the office, you might have a network of people that you bump into and then go for coffee or a lunch with. These conversations are goldmines of unintentional knowledge transfer. When working remotely, whether that’s full time or even part-time, these networks take more effort to maintain and build, but it’s effort that’s well worth spending.

Schedule regular catch-ups with people, just half an hour is plenty of time to keep the relationship alive and gain a nugget of useful information. As well as catching up with people you already know, ask them explicitly who else they think you should talk to? You’re not going to bump into people randomly in a Zoom call, so be more intentional about expanding your network.

If you have information that might be useful to others, then share it. Similar to what I was talking about above, look for others that you can catch up with and give them information. Also in a world full of cross-continent collaboration and multiple timezones written sharing of information is more valuable than ever. If you’re feeling fancy then putting together a newsletter for your department is a great way to spread the word about what you’re working on.

I’m a big fan of the 5/15 update, which is probably a topic for a future post. The summary however is that each week you spend 15 minutes writing a summary of what you’ve been working on, aiming for it to take five minutes for someone to read. I post mine in a slack channel, but you could share it over email or wherever fits with your company culture. This has benefits to me as it lets me look back over what I’ve achieved in the last year, it helps me organise my thoughts on what’s happened in the last week, and I try to commit myself to goals for the following week. It is also useful to others because they can see what I’m working on, what I’m thinking about and generally what is going on in my world - exactly what you might get from a chat over coffee or sitting near me in an office.

Of course, you could argue that intentionality comes with a cost (usually time), and if the pandemic is over why don’t we all just go back to the office where we get this for free?

At the time of writing the pandemic is very much not over, and the Omicron variant is rearing its ugly head, so returning to the office is not desirable for everyone. But probably more importantly, it’s pretty clear that hybrid or even fully remote, teams are going to the default in the future, rather than the exception. Whether it’s just someone who works in another country or the whole team who are remote there is likely to be some degree of remote working, and for a hybrid team, the lack of office-based osmosis is more of a risk than for fully remote teams.

With a fully remote team, everyone is in the same boat. No one is getting the in-person benefits, so everyone needs to make an effort to compensate. With a hybrid team, some people will be benefiting so it’s easy for them to forget that remote workers are missing out. In a hybrid team, it’s crucial that everyone makes an effort to include all team members in discussions and knowledge sharing. While the team lead can share information, that should be a last resort and it’s the whole team’s responsibility to ensure that everyone is included.

Despite some of the challenges of remote working, and remote leadership, this is now my preferred way of working. It’s certainly been a steep learning curve, but with some focus and effort I feel it’s possible to get the best of both worlds. Do you have any tips for sharing knowledge while working remotely? Let me know in the comments below.

Ownership and High Performing Teams

2021-11-05T00:00:00+00:00

Imagine you’re making a change to a codebase you don’t like. Testing is difficult and you don’t have a good understanding of how all the parts of the system fit together. It’s tempting to just do a bit of cursory testing, deploy it to production and keep your fingers crossed. This is not a pleasant situation to be in, and will inevitably lead to production incidents, sad developers and cross managers.

As a manager, if you encounter a situation like this you’ve got a choice to make about how to fix it. Instinctively you might want to put more processes in place around deployment and testing. Maybe kick off a project to refactor the code base and make testing easier.

Maybe you can reduce failed deployments, but at what cost? Slower development caused by extra bureaucracy and more policies to follow is going increase friction, delay releases and eventually lead to an unhappier team.

The underlying problem is that the team feels disconnected from their software. They might own it according to your org chart, but they don’t feel ownership of it. This lack of ownership can manifest itself in many ways, but principally it results in a “good enough” attitude and a lack of interest in the negative effects that a deployment might cause, or future issues caused by the code change.

So, rather than adding more policies how do you encourage the team to feel more like the owners of the software they are responsible for? There’s no one answer to this, but you can start by looking at how detailed their future technical plans for the software are. A team that feels a high degree of ownership will likely know exactly what technical changes they want to make over the coming months or years. A team that doesn’t will be content with just tinkering around the edges, or maybe they have some plans but talk about them in a jokey, “we’ll never get around to this” kind of way.

If the team doesn’t know what they want to do with their software then you as a leader can help by taking them through a process of working out what changes they do want to make. These need to be practical and realistic, this is no time for pie-in-the-sky “let’s just start from scratch” ideas. If the team is already in difficultly then you need to start small, and incrementally make improvements.

Begin by assessing where the biggest pain points are. Once you’ve identified the area come up with some easy wins, and create time for the team to work on them. By starting small and getting something done you can help the team feel a sense of achievement. It won’t happen overnight, but once the team feels like they are making progress they will start to feel a sense of ownership.

Another aspect of feeling ownership is a connection to your clients. How easy this depends strongly on what type of business you’re in, but it’s worth fostering this connection if you can. Maybe people can spend a day shadowing a client? If you run user experience research then developers sitting in on a few sessions will help build that connection. Developers don’t need to build as complete a picture as a product manager or UX researcher needs to, but if you’re building something that it’s not possible to dog food then gaining some first-hand connection with a subset of users will help people to think about them in less abstract terms.

To try and build an understanding of your clients you might be able to get one to attend a town hall or other meeting and talk about what impact your changes have had on them, what their priorities are, or a recent success that your software has enabled. Hearing directly from a client, and putting a face and a voice to them is hard to beat as a way to build empathy and understanding of your users.

If you’ve been successful you should start to notice a change in the behaviour of the team. They might be more proactive about suggesting improvement projects and be keener to iteratively improve on what will still be a “bad” codebase. Hopefully, you’ll notice a keenness to resolve issues more quickly, and more attention to detail when preparing and executing changes in production. If a company’s users become the team’s users then they will feel a personal responsibility for their experience, and for the success of your product that can only be a good thing.

Anti-Consumer Electric Car Charging

2021-10-14T00:00:00+00:00

Recently I went on holiday to the island of Jersey in the Channel Islands. As an electric car owner this currently requires more planning than I would like. Inevitably, running an electric car on holiday is not quite as convenient as being able to charge your car up at home, or being able to pop into a petrol station and fill up in a couple of minutes. There are three 50kw chargers on the island, which can get me from 20% to 90%+ in just under an hour. Given the size of the island a full battery will last for five or six days, so we expected this to be ok.

Although we’ve owned a Tesla for around nine months at this point our only drive longer than the range of the car was to Northern Ireland and back, which necessitated a single stop in each direction at a Tesla Supercharger and was quick and easy, exactly what you’d hope for. While we were there we were able to charge using a three-pin plug where we were staying, so charge was not an issue. Unfortunately, the rented accommodation we were staying in on Jersey didn’t have a plug near the car park, so charging overnight was not an option.

On our first day we happened to park in a car park with some 7kw chargers. We thought why not get a top-up while we buy some food, so we plugged in and tried to work out how to start a charge. Most chargers on the island are run by Charge Your Car so I had already downloaded the app. First impressions of the app are not good - it hasn’t even been updated to support a modern iPhone’s aspect ratio. Still, it was fairly easy to navigate to the right charge point on the map and click the “start charge” button. This then directed us to add a card to the app so we could pay. The first stage worked fine, but then you need to add the CVV number and the app returned a 404 Not Found error and that was it - there was no way to start the charge. After 20 minutes on hold to their helpline, we gave up and left.

The next day I went to the nearest 50kw charger which while also run by Charge Your Car supported contactless payment. It worked perfectly, so I could focus on my holiday.

After running the car down to about 15% I went back to the charger, only this time it refused to detect that our car was connected. While I was on hold to the helpline a nice man from Jersey Electric happened to be passing and he tried to help. He wasn’t able to fix the fast charger so he directed me to a nearby 22kw AC charger. Sadly this was limited to 11kw due to the Model 3’s onboard charger. It was also managed by Charge Your Car so the helpful Jersey Electric man lent me his RFID card to so I could top up. After waiting an hour (which at 11kw only gained me 18%) I decided to risk a different 50kw charger.

After some difficultly finding and then parking at the charger as it was in a very tight hotel car park I was eventually able to connect and set about trying to work out how to start the charge. This charger didn’t support contactless and was run by BP Pulse. Downloading the app and registering was reasonably straightforward, but instead of registering my card, I needed to top up a minimum of £10. Once this was done the charge started and I sat back to enjoy my book. Unfortunately, after less than 10 minutes the charger crashed and rebooted. When it came back it refused to start the charge again. Having wasted most of my morning and now on 50% charge, I headed off to enjoy the rest of my holiday.

Fortunately, for the other two times I used the Charge Your Car rapid charger it worked flawlessly and as expected the Tesla Supercharger on the return journey was perfect. Sadly this experience added more stress than I would like to an otherwise enjoyable holiday. The UK is definitely not ready for the EV revolution, unless you are able to charge at home, or only travel within the Tesla network’s range.

What suprised me most was just how anticonsumer the two charging companies I had to use were. Charge Your Car appears to have been abandoned - an app that’s only had two minor updates in two years, and currently fails in its primary purposed - to let you start a charge at their charge points. We also spent over two hours on hold, and the phone was never picked up. BP Pulse was better, but only marginally. I did manage to start a charge, but the unreliability of the charge prevented it from successfully completing. They also required me to top up a minimum of £10, so I now have over £8 trapped in their app with no obvious way of getting a refund.

Can you imagine if petrol stations worked this way? You needed to download an app for each brand of station, and they forced you to top a minimum of £100 before you could fill up. Half way through filling up it gives up and refuses to add any more petrol. Cars would never have taken off and we’d still be using horse and carts.

As a smug Telsa driver with access to the Supercharger network it would be easy to think that I’m ok. That’s only the case when a Supercharge is en-route though, and while widespread there aren’t charges everywhere and enivatably I’ll need to use third party chargers occasionally.

The UK government seems to be aware of these issues, which is a nice change. They recently published a report which among other things highlights the reliability issues, and the consumer unfriendlyness of requiring app downloads or subscriptions to be able to charge.

In a related move the Competitions and Markets Authority have opened an investigation into a couple of charging companies who have signed exclusive deals with the main providers of motorway service stations. They don’t mention Tesla, but given the exclusivity of the Super Charger network it wouldn’t surprise if they were forced to open it to other users. This is something that has been rumoured for a while, but might be taken out of Tesla’s hands.

In general being an electric car owner is great, but I hope the market and governments pressure charging companies to stop being so anti-consumer, or it really will set the revolution back by years.

Deriving From Untypable Classes

2021-08-06T00:00:00+00:00

There’s going to be a bit of a change of pace this week, compared to my more recent posts. This time we’re going to be getting deep into the weeds of Python typing.

One of the biggest recent changes to Python has been the introduction of type annotations. Back in the dim and distant past I did my third-year university project on type inference in Python, around the time of Python 2.3. Now though, it’s a much more mainstream part of the Python ecosystem. Alongside tools like black and pylint, the type checker mypy is a core part of my standard Python set-up.

Adding type annotations to your code, and integrating a type checker into your CI pipeline gives you many of the benefits of a statically typed language, while retaining most the speed of development that is associated with Python. The dynamic nature of Python, and the fact that type annotations haven’t been widely adopted by libraries that you might depend on, means that type checking has its limitations and sadly this means it might not be obvious when the type checker has exceeded its abilities to detect errors.

Recently I was investigating a CI pipeline failure for a merge request opened by Renovate for Google’s BigQuery Python API library. The failure was in pylint, saying that a type didn’t have the attribute name we were using. At first, this seemed like a simple failure, but after more investigation, I noticed something odd about it.

bin/filename.py:86:38: E1101: Instance of 'LoadJob' has no 'num_dml_affected_rows' member (no-member)

The code it was flagging the error for looked similar to the code below. The error was on the second line, where the result of the query is being used.

query = client.query(sql)
rows = query.num_dml_rows_affected

Why is the error being flagged for LoadJob? We’re running a query, so wouldn’t that be a QueryJob? Checking the library documentation showed that num_dml_rows_affected was still a valid attribute, so the error being raised is spurious. Checking the changes in the upgrade lead me to this pull request. The pull request changes the query function logic from:

def run_query(job_id: int, sql: str) -> QueryJob:
    *snipped*

def query(sql: str) -> QueryJob:
    job_id = get_job_id()
    job = run_query(job_id, sql)
    return job

to:

def run_query(job_id: int, sql: str) -> QueryJob
    *snipped*

def get_job(job_id: int) -> Union[QueryJob, LoadJob, ExtractJob]:
    *snipped*

def query(sql: str) -> QueryJob
    job_id = get_job_id()
    try:
        job = run_query(job_id, sql)
    except BigQueryError:
        job = get_job(job_id)
    return job

The new function call get_job returns a union of all of the different job types, but the original query function only returns a QueryJob. Because the job id refers to the query we’re executing get_job can only ever return a QueryJob, but there’s no way for any static analyser to know that. pylint relies on type inference, and as far as I know doesn’t use the type annotations to calculate return types. pylint infers that the return type of query is now Union[QueryJob, LoadJob, ExtractJob], when you try and use the return value it will only succeed if the attribute is available on all possible types in the union. This explains why the error was saying that num_dml_rows_affected is not available on a LoadJob object - it’s not!

The python-cloud-bigquery project uses the pytypes type checker, but I confirmed mypy has the same behaviour. The change to the library itself doesn’t trigger any errors. It’s only when using pylint on a call to the library that an error is raised.

But why do the type checkers not flag this as a problem? Even if you know this code is correct there is no way they can, and this is precisely the sort of error you use a type checker to try and prevent. There’s something fishy going on, so I tried to reproduce the error using as little code as I could.

from typing import Optional, Union

class A:
    pass

class B(A):
    pass

class C(A):
    pass

def subfunc(arg: Optional[str]) -> Union[B, C]:
    if arg is None:
        return B()
    else:
        return C()

def func() -> B:
    r = B()
    r = subfunc(None)
    return r

Here we have three classes, the base class A and two derived classes. In func we have the variable r which holds a B. We call subfunc which can return either B or C, and assign that to r before returning it. Both mypy and pytypes flag func as having problems - it returns Union[B, C], not the B that was declared.

On further investigation, I discovered that my example code doesn’t quite reflect what’s going on in the library. A is not the base class, it derives from a class from another library. Replacing A with this next snippet of code is more accurate.

import google.api_core.future.polling # type: ignore

class A(google.api_core.future.polling.PollingFuture):
    pass

Making this change causes the errors raised by both mypy and pytypes to disappear. They claim the code is fine when clearly it’s not. The # type: ignore part of the line is the smoking gun here. It’s needed for mypy (and inferred in pytypes) because Google’s api_core library has not yet added type annotations. All classes in this library get treated as the Any type, which means the type check will essentially ignore any expression involving that type. This is normal and what I would expect.

What I wasn’t expecting was how this interacts with class hierarchies. When you derive from an untyped class the derived class also becomes equivalent to the Any type (because the base class is assumed to have all attributes), so all derived types are equivalent, in this case, B and C. This in turn means that no errors are raised when returning Union[B, C] instead of B.

pylint picks this error up as (as far as I know) it doesn’t yet take into account the type annotations and is still relying on pure type inference. While the code in question is bug-free, the automated checks are not able to help in verifying this. When the api_core library is updated with type annotations the type checkers will start raising this error, despite the change to upgrade the library appearing to be unrelated.

The Any type has a very specific use in Python - when you are using the dynamic nature of Python to go beyond what can be represented by the type system. This is something you should be careful about and go into with your eyes open. What this discovery has taught me is that your dependencies might also introduce the Any type into your code, and its effects can filter through in some unexpected ways.

What do you think of type checking in Python? Have you found any edge cases? Let me know in the comments below!

Writing A DevOps Vision

2021-07-09T00:00:00+00:00

DevOps as a concept has been around since around 2010, but implementing the ideas behind it, particularly when you’re in a team that is supporting old monolithic codebases is challenging. For several years we had engineers fulfilling the role of a “DevOps Engineer”. However, we always knew that having a specific person working on DevOps is a bit antithetical to the DevOps concept - it’s supposed to be a state of mind and a set of practices rather than a job role.

The aim was always to have that engineer act as a source of expert knowledge and an enabler. Teams were still supposed to own their code, processes and deployments, but in reality, DevOps related work was often thrown over the wall to that engineer with the expectation that it was their problem, and not the team’s problem.

We ended up in a situation where we had to make a choice - hire a new engineer into the same role, or attempt to spread the work across all engineers. We chose the second option, but that then poses the question of how to change team culture across a department, so that DevOps becomes a standard part of the team’s process, much like Kanban, Scrum or any of the other ways the team organises themselves.

When introducing this change I wanted to ensure three things, that people understood what the scope of the change was, why it was important, and that there was help available to aid them in taking on this responsibility.

Changes such as this need some careful thought before they are rolled out - as well as needing to persuade people that they are a good idea, and to reassure them that they will be supported through the change, you need to make sure the changes actually happen. When asking a team to change their processes there is a significant amount of inertia to overcome, and a real danger that they just shrug their shoulders and say “that’s just management waffling on about nothing important”.

To make sure the new process achieves its goals I followed a three-step process. Firstly I prepared a document describing the change, which covered all the points above and can be referred back to in the future. Secondly, I presented the process during a town hall. This ensured that people couldn’t claim to have not read the document! Finally, I kicked off a project to improve our build pipeline reliability.

The key part of the document is the section that describes the changes I wanted to enact. To make this as clear as possible I structured it as a series of bullet points preceded by “Teams should…” Each bullet point is a short behaviour, followed by a longer piece of explanatory text. To try to ensure the document made an impact these were phrased deliberately stridently, perhaps even controversially, to try and provoke some interaction.

It’s probably too early to declare the change a success or not, but there are encouraging signs. Teams are working well together to align on processes, they are taking responsibility for their builds, and the quality of our pipelines is improving.

So if you want to make a cultural change, then I suggest following these three steps - document the change, present the change, and then start a project to initially force the change you are looking for, with the aim that it will continue long beyond the project itself.

The Power Of Team Dashboards

2021-06-18T00:00:00+00:00

Using metrics and dashboards is a well-understood tool when monitoring the health and performance of software, or your profitability or other key business metrics. What is less common is using the same tools and techniques to monitor the health and performance of the team behind the software. I’m not suggesting using dashboards to report on individual developers, but as a tool to help the team focus on improving their own processes, it can be very useful, provide it’s handled carefully.

My own journey started when I was promoted from a team leader to an engineering manager, responsible for five teams. The change in level resulted in a significantly different view, but also great difficulty in knowing where to focus my efforts. When you’re a team leader you are so close to the team that you hear and feel every change in mood, and have intimate knowledge of all projects and their current state. Suddenly being responsible for five teams gives you a great view to take advantage of areas of collaboration between teams, and removes you from the noise of day to day life so you can focus on the biggest issues. However, it also removes you from the firehose of raw information so it can be hard to know where you should spend your time to get the best return on your energy.

The solution to this problem is the same as it would be for a software system you want to improve, measure everything! Error rates, latency, memory usage, throughput - measuring software has a long history, but measuring a team is much tricker. Firstly decide how to measure, and then what to measure. Start small, and gradually add metrics as you find areas that would be useful to report on.

How to measure is simple, just use whatever technology your company standardises on for dashboards. In my case, I started with a Python script that wrote to a Google Sheet and displayed the data in a Google Data Studio Report. As the report evolved we transitioned to using some company standard and some custom BigQuery databases. Whatever technology you use make sure it’s automated, that it runs regularly and that it produces something that is easily and widely available.

When deciding what to measure start with any team, department or company goals you might have about team performance. In Ocado’s case, there is a company level goal around PeakOn scores so we used a BigQuery dataset to visualise our results. Next, consider what factors you think will affect team morale the most. This might be the number of on-call escalations, the MTTR for incidents, or something else. Lastly, you could look at leading indicators for the team delivery performance.

Tracking metrics that are lagging indicators of performance is common - number of tickets completed, the cycle time of completed tickets, etc. While there is some value in these, they don’t help you in deciding where to focus to help the team deliver. If you spot a dip in completed tickets then it’s too late, the dip has already occurred. You can then react and try to work out why it dipped, but it’s much better if you can prevent the dip in the first place. Some example leading indicators are:

Number of tickets in progress. An increase here can indicate tickets not being completed, but new ones still being started.
Time spent in progress for current tickets. Similar to cycle time, this metric should look at how long tickets that haven’t been completed have been in progress. An increase in this will precede an increase in cycle time and might indicate that developers are struggling to complete or deploy their changes.
Tickets that have exceeded 90% of your cycle time. Rather than a number, this should be a list of tickets that are already guaranteed to increase your cycle time.

Once you have a simple dashboard up and running you need to ensure the data is looked at and reacted to regularly. To do this integrate the dashboard into your leadership team rituals so thinking about it becomes second nature. I check the report every Monday morning and talk to my team leaders about anything that jumps out at me. As a group, we also discuss the cycle time and on-call escalation charts as any change in those could quickly derail our delivery plans. We also have a whole department get together every other week where I talk through any trends and ask the developers to think about what their team can do to improve things, or how I can help them.

When talking to a department of developers you clearly need to tailor your messaging differently to when you are talking to a group of team leads. Focus on areas that are important to the morale of your teams. This will likely be the feeling that they are able to work quickly (i.e. cycle time) and how much time is spent dealing with support issues.

It’s particularly important when talking to a room of developers to treat your dashboard as a guide, and not a stick. Definitely do not use it to evaluate individual developers, and try to avoid the temptation to compare teams against each other. Every team has its own set of challenges, and what’s important is that everyone is learning and getting better, not who is “winning”. It’s very useful to have the ability to filter your dashboard to an individual team but avoid adding the ability to include multiple teams at the same time.

Earlier I mentioned including a list of tickets that have exceeded 90% of your usual cycle time. This will produce a list of tickets with issues and are therefore worthy of extra attention, but highlighting them during team meetings can be dangerous. It’s very easy for developers to treat a ticket being called out for taking an unusually long time as an attack on them or their abilities. This should not be the case, and it should be an opportunity for the team to discuss how they can improve to get this ticket completed and to prevent similar tickets from stalling.

To allow teams to deal with long tickets you shouldn’t call them out during larger meetings. Definitely do not include assignee names on any dashboards, instead just include ticket numbers, titles and team names. Team leaders should use the report to steer discussions towards problematic tickets during standups or retrospectives.

I have found having a team dashboard an invaluable tool that has made me a better, more proactive manager. If you start small, iterate, and embed your dashboards into your team rituals from the start you’ll quickly see how you can focus your energies where it can have the most impact.

Let me know what metrics you use, and how dashboards fit into your rituals in the comments below!

Replacing Travis CI With BuildBot

2021-05-28T00:00:00+00:00

Back when I reactivated this blog I posted about using Travis CI to automate the build process. Sadly at the end of last year Travis announced they were ending free builds for all public repositories, and only authorised open source projects will now get free build credits.

The repository for this blog is publically accessible, partly in case anyone wants to see my draft posts, or raise a merge request to fix a typo, but mostly because why not? That previously allowed me to not worry about the cost of building the site, but it’s not unreasonable for a private company who need to make a profit to want to focus their generosity on actual open-source projects. I certainly don’t blame them for the policy change, although I hope the approval process for open source projects is easy and widely applied, so it’s not just a few big projects that can take advantage of it.

There are many other build services out there, but I don’t really want to pay to build my blog, so I focused on finding an easy to use open-source CI system I could run locally. In the end, I settled on BuildBot which has been around since the early 2000s, and I actually have some experience with it from around that time. BuildBot has the advantage of being Python-based (one of my preferred languages) and it uses Python as its configuration language so setting it up should be straightforward.

I already have a number of Docker containers running on my Synology NAS. Adding Buildbot requires adding two more, a buildmaster and a worker. The build master needs no special handling, you can run the standard Docker image and just mount a configuration file into the right location. For the worker, it needs a little more thought as you need to bake any dependencies into your image so they’re available to the builds. It is relatively straightforward to build on top of the default worker image, and my Dockerfile can be found here.

One problem I encountered was that the Buildbot documentation is focused on the model of one project that has one BuildBot instance, but I need to share the builder across multiple projects. Fortunately, it works well if you have a SingleBranchScheduler per project, and use a ChangeFilter to filter the incoming webhooks from GitHub.

c['schedulers'].append(schedulers.SingleBranchScheduler(
                            name="all",
                            change_filter=util.ChangeFilter(
                                        project='andrewjw/site'),
                            treeStableTimer=None,
                            builderNames=["site"]))

Each scheduler points to a different builder, which uses a unique BuildFactory to specify the build steps. I try and keep my configuration in a buildbot.sh within the projects themselves.

factory = util.BuildFactory()
env={"GEM_HOME":"/home/buildbot/.gem/ruby/2.7.0"}
factory.addStep(
    steps.Git(
        repourl='git@github.com:andrewjw/site',
        mode='incremental'))
factory.addStep(
    steps.ShellCommand(
        command=["bash", "./buildbot.sh"],
        env=env))

One advantage of Travis was that much of the build process was automated, and you didn’t need to configure it. With BuildBot you need to specify every step, and the running of code only on master, or only on a tag is not as simple. After much trial and error, the following code works. You can see the full file in the glowprom project. First, we extract the branch name of the current checkout, then compare it to master or use a prefix match to determine if we’re on a tagged version. When running on master the Semantic release call is the final step, so we know all the tests have passed. This will tag a release, which again triggers the pipeline, but this is a tag with a v as the initial character, so we build a Docker image and push it to the Docker Hub.

BRANCH=$(git rev-parse --abbrev-ref HEAD)
echo "Building branch $BRANCH"
if [[ "$BRANCH" == "master" ]]; then
  COVERALLS_REPO_TOKEN=$GLOWPROM_COVERALLS_REPO_TOKEN coveralls
  semantic-release publish
fi
if [[ ${BRANCH:0:7} == "heads/v" ]]; then
    ./docker_push.sh
fi

Timeliness of feedback on your commit is an important factor in enabling quick developer cycle times. Travis enabled that by using a webhook so it was notified immediately when you pushed a commit. The problem with running BuildBot internally is that it isn’t externally accessible. BuildBot makes it easy to run a scheduled build, so you can build every day, hour or even more frequently if you want. The problem is that you have to wait for that build to be triggered to get feedback on your commit, or you could manually trigger it, but who wants to add a manual step like that?

Fortunately, I already host this blog on Linode, and I have a VPN set up between the server and my internal network. I added the below config to the Apache instance that serves the blog. For my external DNS name it uses mod_proxy to forward requests to /change_hook/github on to the BuildBot instance, which will go via the VPN. GitHub and BuildBot already supports a shared secret for authentication that the request is valid. This is set up when you add the webhook to the GitHub API.

<VirtualHost *:80>
    ServerName external.dns.name

    <Location /change_hook/github>
        ProxyPass http://internal.dns.name:8010/change_hook/github
        ProxyPassReverse http://internal.dns.name:8010/change_hook/github
        <Limit POST>
            Order allow,deny
            Allow from all
        </Limit>
    </Location>
</VirtualHost>

Requiring a remote server and VPN adds a significant amount of complexity to the setup. I believe most home routers support some form of port forwarding, so you could probably also use that. However, I would be concerned about exposing more of my home network to the internet than necessary. Please let me know in the comments if you have any suggestions on how to simplify the setup.

This setup has been running for a couple of months and seems to be working well. It is both reliable and means I can sleep easy, knowing I’m not leeching of a company who were just trying to help real open source projects.

Scheduled SMART Checks

2021-03-10T00:00:00+00:00

For years hard disks (both spinning rust and SSDs) have had a built in monitoring system that tracks various metrics about the health of your disk, called SMART. In the old days if you were lucky you might get some warning that your disk was about to fail because it would start to make a nasty noise. In the modern era of SSDs you likely won’t get any warning, and suddenly boom, your laptop won’t boot or mount the disk.

Obviously nothing is perfect, and any monitoring can miss a failure, but the potential of some warning is better than definitely not getting any. Also this is no subtitute for a proper backup and recovery strategy, but in most home situations people don’t have spare laptops or hard drives just sitting around.

It would be relatively easy for operating system vendors to automatically detect SMART capabable drives and automatically run a check every so often. If it fails, they could pop up a warning about a potential imminent failure. As far as I know though, no-one does this.

There is a simple command line tool that lets you interrogate SMART attributes yourself, smartctl. It is available in all Linux distribution package repositories, and in both Homebrew and MacPorts for Mac OS. I’m not a Windows user, so I’m not sure about using it there.

On both my work and my personal MacBooks, and my home Linux server I run something similar to the following. First I have a script which enables SMART monitoring, then triggers a short test, waits for it to complete and finally emails me the output.

#!/bin/bash

/opt/local/sbin/smartctl --all /dev/disk1 -s on

/opt/local/sbin/smartctl --all /dev/disk1 -t short 2>&1 > /dev/null

sleep 600

/opt/local/sbin/smartctl --all /dev/disk1 | /usr/bin/mail -s 'MacBook SMART' myemailaddress

On Linux it’s easy to use cron to trigger this script weekly. On Mac OS it’s a little more complicated, but the following plist file will do it. Simple place it in /Users/username/Library/LaunchAgents/smartctl.plist (replacing your username and script file name as appropriate), and it’ll get run weekly, when you switch your laptop on.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Label</key>
        <string>smartctl</string>
        <key>Program</key>
        <string>/Users/andrew/bin/smartctl</string>
        <key>StartCalendarInterval</key>
        <dict>
            <key>Hour</key>
            <integer>0</integer>
            <key>Minute</key>
            <integer>0</integer>
            <key>Weekday</key>
            <integer>1</integer>
       </dict>
       <key>AbandonProcessGroup</key>
       <true/>
    </dict>
</plist>

On my work laptop smartctl is reporting PASSED, but one attribute is failing. I guess we just need to wait and see what happens. Fingers crossed it holds out until I can get Apple-silicon-powered MacBook Pro.

173 Wear_Leveling_Count     0x0032   096   096   100    Old_age   Always   FAILING_NOW 12309581794157

Working In A Crisis Is The Easy Part

2021-02-05T00:00:00+00:00

Recently one of my team asked me how they were doing, and to be fair I think they’ve been doing great. The challenge though is that they started as team leader right about the time that COVID-19 hit. Working for an online supermarket, around March 2020 was definitely challenging, but in some ways it actually made things easier. When there is a fire to put out, it’s easy to know what to do next - you put the fire out! And after that? You put the next fire out!

Certainly, there are challenges for being a team leader during a crisis. Perhaps people in your team are stressed or losing their cool. You might have stakeholders shouting at you to fix things quicker. Maybe you have to take shortcuts that you know lead to technical debt, but that will get things fixed quicker.

Calming people down, and protecting your team are important skills for a team leader. Knowing when to make pragmatic technical decisions is something all developers should know. And recognising when your stakeholders are crying wolf and you can push back on fixing their fire is important. For us, back in March, it was pretty clear that this was a real crisis and we needed to do all we can to help.

Things have moved on since then, and although the world is still suffering, from the perspective of us writing software to help keep people fed, things are pretty normal. The challenge for my new team leader starts now.

With a quieter backlog things become much more uncertain. What is the right thing to work on next? How much time should we spend tackling technical debt? What direction do we want to move in with our architecture? How do you keep your team motivated? All of these are much harder to answer when there isn’t a stakeholder shouting at you to fix something.

The key thing to remember is that quieter times give you more time to think, but that you need to carry your team with you while you do it. Set up meetings to discuss architecture and tech debt, and use them to drive agreement amongst your team about what their priorities for improving your software are. Once you have broad agreement on how you want your software to evolve, it becomes easier to decide what to work on next. Can you deliver this change while also delivering customer features, or do you need to pause, refactor and then delivering customer features will become easier?

When assessing performance it’s important to remember what context you’re working in, and what it would be reasonable to achieve. If things are on fire then putting those fires out is a good achievement, but if things are calm then you should judge yourself against more traditional measures of success, such as the delivery of features, team alignment and the quality of your roadmap.

Meter Readings Over MQTT

2020-12-02T00:00:00+00:00

In a previous post I talked about swapping the in-home device (IHD) supplied by my electricity and gas company for one produced by Glow. This connects over wifi and gives access to the raw data coming from your smart meters over MQTT. To collect this data and integrate it into my house monitor I needed to listen to the MQTT topic and expose the data to Prometheus.

This was quite similar to my previous project to expose statistics from my router, but as this was processing JSON data being exposed over MQTT it was a lot simpler than parsing raw text over SSH. As before I created a GitHub repo with my usual Python linting, testing and build scripts set up.

Connecting to MQTT is straightforward, using the Paho MQTT library. When they’ve set up your MQTT account Glow’s support team will supply you with a topic name that you can use, along with the username and password you set, to connect. The code below shows how to do this. We’ll cover the on_message callback next.

def connect(args, on_message):
    client = mqtt.Client()
    client.on_connect = functools.partial(on_connect, args.topic)
    client.on_message = on_message

    client.username_pw_set(args.user, password=args.passwd)
    client.connect("glowmqtt.energyhive.com", 1883, 60)

    client.loop_forever()

As you can probably guess, the on_message callback is called every time you receive a message on your MQTT topic. In our case, this will be the raw ZigBee data encoded as a JSON object. This GIST has a good summary of the available values, so I won’t repeat them here. The details for the electricity meter are included under elecMtr then 0702. There are a series of values for daily, weekly and monthly usage, but the most useful is the raw meter reading. We can extract this like so:

elec_multiplier = int(elecMtr["03"]["01"], 16)
elec_divisor = float(int(elecMtr["03"]["02"], 16))
elec_meter = int(elecMtr["00"]["00"], 16)

electricity_meter = elec_meter * elec_multiplier / elec_divisor

The raw meter reading sent a hex-encoded integer, which is scaled by the given multiplier and divisor. Why it’s like that I’m not sure, but I do appreciate Glow providing as low-level data as they can, even if it requires us to deal with some oddities like this. Once you’ve decoded and scaled the meter reading we can expose the metrics to Prometheus in the usual way.

The queries I’m using in Grafana are as follows:

rate(meter{type="electricity"}[5m])*1000*3600

increase(meter{type="gas"}[30m])*10

The electricity query is multiplied by 1000 as the raw data is in kilowatts, and the graphs look better when Grafana converts from watts in kilowatts itself. rate returns the per second rate, but as the number we want is kilowatts per hour we multiply it by 3600. The gas query is simpler - as the data is sent only once per half hour we look at the increase in the last 30 minutes. The raw data appears to be in tens of cubic meters, so we multiply by 10 to get the right value.

The graphs above show my usage electricity over the course of a day, and gas over a week. In the electricity graph, you can clearly see when the oven was on (2 pm-6 pm) and when I was charging my car (3 pm-5 pm). Other spikes during the day are the kettle or toaster being used. The peak right at the end is our dishwasher starting. Even in the middle of the night, our fridge running is clearly visible as a series of regular peaks. The gas data is much noisier, although you can see when we get up and go to bed. I think a longer analysis of gas usage will probably be more interesting.

Clockwise

2020-11-18T00:00:00+00:00

As soon as you make the leap from developer to manager one of the most immediate changes you notice, apart from the crushing uncertainty of not knowing what it is you’re supposed to be doing, is that your calendar will fill up quicker than you can say “let’s schedule 30 minutes to discuss that”.

Clockwise is a new tool that has proliferated around the managers at work which claims to solve your meeting woes by taking over the scheduling and booking rooms for you. It’s like having your own PA, only without, you know, actually being important enough to justify one.

Technically Clockwise is a browser plugin which integrates with Google Calendar. Every day at 4pm it looks at meetings occurring over the next week and rearranges them to resolve conflicts, and to try and create blocks of “focus time”. Focus time is a chunk of meeting-free time, two hours or longer, in which you can get stuck into work, rather than having a series of 30-minute gaps where it’s hard to achieve anything.

Clockwise will only move meetings that you’ve enabled it for, so you don’t have to worry about it completely messing up your schedule. You also don’t need all participants in a meeting to have Clockwise installed. It can move any meeting you’re invited to and will try to optimise the scheduling for everyone. There are some benefits to having Clockwise installed though - you can configure it to prefer focus time in the mornings or afternoons, and Clockwise can manage a lunchbreak for you too.

In general Clockwise works great. Although it is impossible to create much “focus time” given my schedule, it does a good job of automatically resolving conflicts and creating focus time where it can. There are a few niggles though.

Sometimes a meeting will not be moved, and you’ll notice that you have an unresolved conflict. It’s not clear why this happens, as one or both of the meetings are enabled for Clockwise. Perhaps more clear errors when Clockwise fails to resolve a conflict would be good.

Sometimes meetings seem to be moved for no reason at all. I have a couple of one to ones that this is common on, and the moves don’t resolve a conflict and don’t create focus time but Clockwise moves the meeting just because it can. Each day in the run-up to the meeting it will ping pong around the calendar, making it hard to remember when the meeting is happening…

Lastly Clockwise doesn’t consider different weeks when moving a weekly meeting. You can configure meetings so they are moved within the week, or within the day. I have all mine set to within the week as this gives Clockwise the most flexibility. Unfortunately, if you have a weekly meeting you can end up with a meeting on a Friday, and then one on the following Monday. It would be good if Clockwise could try to keep at least four working days between instances of the weekly meetings.

Clockwise is predicated on the fact that meetings are movable, and the more meetings you enable Clockwise on the better it’s able to do its job. In Rands In Repose’s classic article on one to ones the first rule he states is:

Same time each week. When you become a manager of people, an odd thing happens. You’re automatically perceived as being busier. Whether you are or not is irrelevant; folks just think you are. Consistently landing your 1:1s at the same time on the same day is a weekly reminder that you are here for them — no matter how busy.

This is the opposite of Clockwise’s goal. Currently, I have all my one to ones enabled, so they do bounce around the week a fair bit, but I find this makes them more likely to happen because they will avoid clashes with unskippable meetings. If you’re important enough to move a meeting with 20 other people, then perhaps you can keep your one to ones static. This is probably the one part of Rand’s advice that I deliberately don’t follow - keeping the meetings flexible makes them more likely to happen every week.

It’s not yet clear what Clockwise’s business model will be. Currently, it’s available for free for everyone, but presumably, that won’t always be the case. For now, though, it’s hard for me to imagine managing my calendar without it.

Glow Bright

2020-11-04T00:00:00+00:00

When my electricity provider, nPower, upgraded me to a smart meter I was excited about getting access to more data about my electricity and gas usage. Unfortunately, it turned out that the extra data provided by the connected meters was only available to the power supplier, and not to me as a consumer. They provided a little device with a screen (known as an IHD, or in-home device) which displayed the current and daily/weekly/monthly usage, but little else. No mobile app, no ability to dig into your historical usage, and definitely no API access.

While I was disappointed and frustrated about not having access to what I consider be my data, I left it as I had more important things to worry about (i.e. kids). With my recent project to look at collecting more data from my house, I wanted to revisit this, and collect the data.

There are two meter standards in the UK, SMETS1 and SMETS2. SMETS1 is the older standard, which is no longer being rolled out. SMETS2 is the current standard, which operates quite differently to SMETS1. The main benefit being that if you switch provider your smart meter will continue to work, something that wasn’t true with the earlier standard. If you’re unlucky enough to have a SMETS1 meter you can ask for it to be upgraded. Fortunately, I had a SMETS2 meter.

The way these meters work is that both the electricity and gas meters broadcast their readings locally over a ZigBee network. This is picked up by your IHD, and displayed live. The electricity meter also listens to the gas readings and every so often uploads them over a mobile phone connection to a central data broker. This is then forwarded on your supplier. smartme.co.uk has much more detail on this process.

The key point is that your data can not only be forwarded on to your supplier, but you can grant access to other companies too. Enter Glow who act as what is known as DCC Other User and use smart meter data to help companies do research and development. They also provide more tools for consumers to use their data.

After installing the app you need to go through a security verification process. This involves uploading details of your electricity meter and address (by taking photos of your bill), a photo of some photo id, and a selfie video of yourself saying “I’m My Name, and I accept the terms and conditions”. After that, your details are sent away for a manual verification process, which in my case took two days. I also received an email asking whether I was planning to buy their display. When I said yes they said they wouldn’t process my application until it arrived. This was a bit frustrating as it meant I couldn’t use the app in the meantime. Once the device arrived the application was processed quickly.

The device itself works ok but does have a few usability issues. For example, the graphs could be designed better. As the example below shows, it doesn’t always pick the best axis.

That’s a relatively minor point though, and generally, it does what it’s supposed to do. I’ve had one occasion so far when it lost its WiFi connection, but a quick reboot sorted that out. The main benefit comes from the fact that you can access your live meter data over MQTT, and I’ll talk about that in my next post.

Bad Words In A URL

2020-10-28T00:00:00+00:00

Today I wanted to share one of the more interesting debugging experiences that I’ve had. This happened quite a few years ago when I was involved in migrating a set of websites over to use a single login. The idea was that as soon as you landed on a site and your session wasn’t logged in, you would be bounced over to an authentication site, which would bounce you back again. The two sites communicated via a backchannel, and if you were logged in on the authentication site the main site would log you in too. If not then you’d browse as a logged-out user, and when you logged in you were bounced over to the authentication site with your credentials passed via a backchannel. If successful then you were logged in on both sites, otherwise an error was displayed.

The backchannel communication was all encrypted with preshared keys, and when the user was bounced to the authentication site they were also given an encrypted token to ensure that a bad actor couldn’t attempt to hijack another user’s session. The exact details of the token aren’t important, but they included the user’s session-id, details of the site they landed on, and the time they were bounced (to prevent against replay attacks).

Everything was working great in testing, and we gradually rolled the change out to more and more users. Eventually, we started getting reports of a small number of users not being able to log in. We were able to determine that they landed on the main site ok, and were bounced to the authentication site, but never arrived there.

A considerable amount of back and forwards with the clients ensued, trying to determine what caused the issue. What made it even stranger is that it was only intermittent. It would start or stop working seemingly at random. For some companies, it worked fine for everyone, but for others, everyone would have an occasional problem.

I don’t remember how we discovered the cause, but it was due to the company’s proxy blocking URLs that contained a “bad word”. We weren’t intentionally sending naughty words, but what we were doing is base 64 encoded essentially random data. Because base 64 encoded uses all the normal alpha-numeric characters, if you generate enough strings eventually you’ll get some English words included in the output (see the infinite monkey theorem).

The solution was to add a random number to the beginning of our encoded data and then test the output against a list of bad words. If it matched we tried a different number, until we got a “clean” output. Once this was deployed users behind the proxy no longer had trouble logging in.

So, the lesson is that if you’re including data in your URLs, particularly encoded random data - think about whether you could possible get blocked. With enough users, someone will be using a strict proxy and will hit that edge case.

Router Stats To Prometheus

2020-10-21T00:00:00+00:00

I’ve previously written about my plan to collect much more data about my house. In the current work-from-home environment the quality of our internet connection is paramount, and I wanted to be able to monitor it and potentially be alerted to any degradation before it becomes an issue.

Although I’ve replaced my wifi with a UniFi based system, I still use the router that was supplied by my ISP - which is a ZyXEL VMG1312-B10D. Like most networking equipment the ZyXel supports SNMP which is a technology for reading and writing stats and configuration from equipment, and aggregating them together. On paper it sounds great, but unfortunately SNMP is a nightmare to work with, and you need a mapping file for each device, which doesn’t exist for this model. After looking into creating this mapping, and integrating my preferred technology slack of Grafana and Prometheus, I decided to change tack and extract the data myself.

Fortunately the router UI contains some plain text data which looks easy to scrape. So, filled with confidence that this would be an easier approach that learning SNMP I spun up a GitHub project and got to work cranking out some code.


============================================================================
    VDSL Training Status:   Showtime
                    Mode:   VDSL2 Annex B
            VDSL Profile:   Profile 17a
                G.Vector:   Disable
            Traffic Type:   PTM Mode
             Link Uptime:   1 day: 4 hours: 28 minutes
============================================================================
       VDSL Port Details       Upstream         Downstream
               Line Rate:      7.881 Mbps       39.998 Mbps
    Actual Net Data Rate:      7.853 Mbps       39.999 Mbps
          Trellis Coding:         ON                ON
              SNR Margin:        5.7 dB            7.3 dB
            Actual Delay:          0 ms              0 ms
          Transmit Power:      - 2.6 dBm          11.4 dBm
           Receive Power:      -20.8 dBm         -11.0 dBm
              Actual INP:        0.0 symbols      55.0 symbols
       Total Attenuation:       18.2 dB           22.4 dB
Attainable Net Data Rate:      7.853 Mbps       47.093 Mbps
============================================================================
      VDSL Band Status    U0      U1      U2      U3      D1      D2      D3
  Line Attenuation(dB):  7.2    40.1     N/A     N/A    18.1    50.3    77.6
Signal Attenuation(dB):  7.2    39.9     N/A     N/A    20.0    50.1     N/A
        SNR Margin(dB):  5.5     5.7     N/A     N/A     7.3     7.3     N/A
   Transmit Power(dBm):-13.7   - 3.0     N/A     N/A     8.9     7.8     N/A
============================================================================

The first step to implement this was to investigate how the built in UI requests this data. An early discovery was that when accessing using HTTP responses are encrypted with AES and then decrypted in Javascript. When accessing with HTTPS (which uses a self-signed ZyXEL certificate) responses are in plain text.

Unfortunately, but perhaps not unsurprisingly, at this point I hit what appears to be a bug in the router. The stats work well for a few hours, but then the router stopped responding. Even more strangely after this occurs it was impossible to log in manually to the web UI - the router responded with a username or password is not valid error. Seemly the only solution was to reboot the router. Extending the Prometheus scrape interval extended the time that the stats worked for, but eventually the same error reoccurred.

Given there’s no way to see what was happening inside the router, and therefore it’s unlikely I could work around the bug a different strategy was needed. Digging around the router UI revealed that you could turn on ssh access. Connecting to that gives you a simple shell, that seems to replicate most of the web UI. The commands xdslctl info and ifconfig return information about the speed of the internet connection, and statistics about the number of bytes sent and received.

ZySH> xdslctl info
xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason:  1
Last initialization procedure status:  0
Max: Upstream rate = 7987 Kbps, Downstream rate = 46854 Kbps
Bearer: 0, Upstream rate = 7987 Kbps, Downstream rate = 40000 Kbps
Bearer: 1, Upstream rate = 0 Kbps, Downstream rate = 0 Kbps

The excellent Paramiko library makes it easy to connect to an SSH server, and despite a few problems caused by the fact that ZySH is clearly ZyXel’s own implementation of a shell, and it doesn’t quite work how Paramiko expects, it was simple enough to get the data returned by these commands into a string that I could parse. A few regular expressions later and it’s turned into a set of Prometheus metrics. Using the built in Python HTTP server makes it easy to serve them up for Prometheus to scrape.

The last step is to create a Dockerfile, which is only a few lines long. It installs the library from PyPI and then runs the app in daemon mode when started.

Docker expects arguments to be provided using environment variables, but the Python argparse library doesn’t make it as easy as perhaps it could to accept argument either on the command line, or by environment variable. To work around this I needed to write some code to check the environment variable if the equivalent command line option wasn’t set. Once that was in place I could configure the relevant password using the Docker Compose env_file option, which reads environment variables from a file.

  zyxel:
    image: andrewjw/zyxelprometheus:0.5.2
    container_name: zyxel
    ports:
      - 9101:9100
    env_file:
      - ./secrets/zyxel

Using the SSH interface appears to be much more reliable the than the web UI, and has been running for several weeks with no issues at this point. I’ve currently not got any alerts configured (no point in emailing if the connection is down), but next time there are complaints about slow internet at least I’ll have the statistics to investigate.

At the moment the code is specific to my brand and model of router, so it’s not that widely useful. If your router has an SSH interface, feel free to open a pull request with support. I’d love to support more routers, even non-ZyXel ones.

House Measurements

2020-10-14T00:00:00+00:00

For a long time now I’ve tracked the weather outside my house with my weather station. I also have smart electric and gas meters which display my usage on a little screen in my kitchen, but I didn’t try to do anything useful with that data. Recently I brought an electric car and given that it’s essentially a giant iPad on wheels it inspired me to look into what data I could collect from it, and from elsewhere in my house.

Towards the end of last year, I upgraded my Synology NAS to a newer model which has an Intel, rather than MIPS processor, partly because it was old and I was worried about it dying, but mostly so I could run Docker containers on it. I’ve been running both a Ubiquiti UniFi Controller and PiHole since then, but I knew as part of this project I’d want to run many more containers so I took the opportunity to tidy up the setup.

Docker Compose is a tool that sits above the normal docker command and it lets you run multiple docker containers while simplifying the management of images and the options you need to set for the container to work correctly. You can find my docker-compose.yml file here.

At work we make extensive use of Prometheus for recording our metrics and Grafana to display them, so they were the obvious choice for my home monitoring as well. Both are incredibly easy to set up, with Docker containers already available.

One key metric I knew I needed to track was the amount of free space on my NAS. I’m a long time MythTV user, and my kids record vast quantities of Hey Duggee which I need to keep an eye on. Fortunately, the Prometheus team makes this easy by providing node-exporter, which produces all the metrics you might want to monitor on a server, including disk space. If you run it inside Docker container then by default it will give you stats from inside the container, but by mounting some key directories inside the container you can get metrics for the whole machine.

  node-exporter:
    privileged: true
    image: prom/node-exporter:v1.0.1
    container_name: node-exporter
    restart: always
    network_mode: host
    ports:
      - 9100:9100
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
      - /volume1:/volume1:ro
    command:
      - "--log.level=debug"
      - "--path.procfs=/host/proc"
      - "--path.sysfs=/host/sys"
      - "--collector.filesystem.ignored-mount-points"
      - "^/(rootfs/)?(dev|etc|host|proc|run|sys|volume1/@docker)($$|/)"

Above a certain level, I don’t particularly care about the absolute amount of disk space, what matters is the direction it’s trending. To highlight when I need to look at the amount of TV being recorded I set an alert to trigger when we dropped more than 20Gb of disk space in a week. The rules.yml file below includes an expression to check the delta of the free space over the last seven days. Connecting an alertmanager instance means I get an email alert which will prompt me to delete any recordings we don’t want any more.

groups:
  - name: Alerts
    rules:
      - alert: NASDiskSpaceChange
        expr: delta(node_filesystem_avail_bytes{mountpoint="/"}[7d])/1024/1024/1024 < -20
        for: 1h
        annotations:
          title: 'NAS Disk Space Change Too Negative'
        labels:
          severity: 'low'

I mentioned that it was getting an electric car that triggered me to think about how much data I could collect from my house. To track my car statistics I’m using the excellent TeslaMate. They provide excellent documentation about getting started with Docker Compose, so I won’t repeat that here. One change I did make was to not use the provided Grafana image, as I wanted to make my own dashboards. Instead, I use a vanilla container and use Grafana provisioning to deploy the TeslaMate dashboards. I’ve checked out their repository on my host computer and mounted the dashboards directory inside Grafana. Using a provisioning yml file will then automatically include the dashboards in Grafana instance, along with any custom ones that are defined.

There’s not much to say about TeslaMate, it was extremely simple to set up and although I’ve not had my car for long, has worked perfectly so far.

While tracking server metrics is a solved problem, tracking other household data is less common. In future posts, I’ll talk about tracking data from my router, our weather station and our electricity and gas usage.

Two Dimensional Dates

2020-10-10T00:00:00+00:00

Last week the UK government announced more than 12,000 cases of COVID-19 - more than double the number from the day before. This increase was accompanied by the following message on the Government’s Data Dashboard.

Due to a technical issue, which has now been resolved, there has been a delay in publishing a number of COVID-19 cases to the dashboard in England. This means the total reported over the coming days will include some additional cases from the period between 24 September and 1 October, increasing the number of cases reported.

This turned out to be an issue with the reporting of positive tests caused by a limit in Excel. An interesting part of the story is the way it was initially reported in the media. Their focus is on the number of the cases reported each day, which due to the delay in processing tests is not accurate normal, but with this delay is a pretty meaningless number. The key measure used to make important decisions is the rolling average of new case over the last seven days, and with this number you can’t at a glance know how the average is changing.

What is particularly odd, is that it turns out that the government do publish the number of positive cases by the date the sample was taken. It’s just that for the last few days the media narrative has been “huge number of cases”, even though that’s largely an artefact of the old incorrect data. Sure they’re high, but they went up a week before, not last Saturday.

Now, I’m not a virologist, a sociologist or a politician. I am a developer, and this got me thinking about how to design data structures for time-series data. It’s tempting to think of time-series data as consisting of a pair of values - the metric, and the date the value applies to. In the example above this would be the number of positive tests, and the date the sample was taken. As we’ve seen so clearly over the last few days this time-series is not immutable, and the history can and will change as new information becomes available.

A better way to think about the data is as a triple - the metric, the date the value applies to, and the date we knew about the data point. Depending on what you want to do with the data the second date could be part of the primary key. In many cases you only ever want to know what is our best view of the history - in this case just having a last_modified date is probably enough. Just having a date that represents when the value was last updated shows that the history is mutable.

In some cases though, you want to be able to ‘re-run’ history, and in that case, making the date_known part of your primary key gives you a two-dimensional view of the data. On one axis you have the date the value applies to, and on the other, it’s the date that we knew about that value. This allows you to ask on this date, what was our view of the data? If you’re training a machine learning model or trying to do any sort of prediction, it’s crucial that you don’t train it on data that you didn’t know at the time. It’d very easy to think you can predict the stock market if you train your model on a complete history of shares prices.

Taking the COVID data as an example, the media seem to be relying on an immutable time-series of the difference in the total number of cases each day - this is an immutable series of (date, value) pairs. The Government’s Data dashboard shows the number of positive tests by day the sample was taken - this is a mutable series of (date, value, last_modified) triples. The most flexible data structure would be an immutable series of (date, date_known, value) triples. This lets you generate both the media’s and the Government’s data, as well as other questions such as how long is it taking on average for a test to be reported? Or, how much higher is the final value compared to what was initially reported?

Timeseries data is all around us, and it’s easy to think that history doesn’t change. Unless you’re measuring something directly then all you have is your best view of the data at a given point in time. Expecting that to not change is both likely to be incorrect, but will also prevent you from asking interesting questions of your data. So the next time you’re building a time-series, try to think in two dimensions.

Is This Thing On?

2020-09-30T00:00:00+00:00

tap, tap, tap Hello, is this thing on?

I started this blog way back in 2008 and was fairly active through to 2012. Unfortunately, apart from a period in 2017, it has been dormant since then. Working for a hedge fund, who are notoriously secretive, and having children, who are notoriously good at sucking up all your free time, meant that blogging really wasn’t an option. Now I work for a more open company (Ocado Technology) and my kids are a little older, so require a bit less time, I’m hoping to resurrect my blog.

Originally I created the blog on Wordpress.com, which worked great. It was free, reliable, and easy to use. Unfortunately, when I started writing a new post the editor had become unusable for me. Perhaps I’m an old fuddy-duddy, but I just want to be able to write my text, add some simple pictures, and have the editor get out of my way. Sadly, it quickly became clear that WordPress was going to get in my way, so it was time to give the blog a new home and a new lick of paint.

For many years I’ve had a Linode server, which I use to host a few small websites. I’ve always been a fan of Markdown, so a static site generated by Jekyll seemed like the obvious choice.

Of course, the only way to start a new project is to buy a domain name. After trying some variations on my name I discovered that theandrewwilkinson.com so I snapped it up. I don’t think I can claim to be the Andrew Wilkinson. I’m not even the most successful Andrew Wilkinson in tech, that has to be this Andrew Wilkinson. However, it beats a wordpress.com address, so it’ll do.

Jekyll was surprisingly easy to get started with. I was a bit worried as a bit of Ruby-phobe, but even for someone who doesn’t know anything about Ruby it was easy to get started. Getting my old posts out of WordPress was straight forward, and they were easily converted into Markdown. I spent quite a bit of time tidying up the generated Markdown, fixing broken links and images, etc, but fairly quickly my new blog was full of my old content.

One advantage of WordPress is that you can hit the big Publish button and your post is live on your site. Fortunately, in the 12 years since I started this blog, many things have come along to make automating publishing a static site easier. Obviously, the first step was to set up a GitHub repository. I have had good success with Travis CI in the past, and it worked well here too. They provide a prebuilt Ruby image, which includes rsync. Running the Jekyll builder and then rsyncing the generated content over to my Linode server on success was really quite simple to get running.

A key part of being a successful blogger (so I’m told) is to keep posting content regularly. WordPress helps support this by letting you schedule posts for the future. This way you can have a queue of completed posts, removing the panic of needing to complete a post by a certain date. Jekyll supports future dated posts, so we just need to be able to trigger the deployment pipeline automatically to cause them to be deployed. Travis CI makes this easy thanks to its support of cron jobs. Just click More Options then Settings for the site’s build pipeline, then towards the bottom is a Cron Jobs section where you can schedule the pipeline to be run daily.

So far everything had gone a lot more smoothly than I’d expected. Converting my old posts to tidy Markdown took longer than expected, but it was more laborious than difficult. The final thing I wanted to tackle before going live was comments.

Initially, I was quite taken by StaticMan. This service works by providing an endpoint you can point a form to, and it will create a branch in your GitHub repo where it’ll add the comment in the right format for Jekyll. Clicking a button in GitHub to merge the branch will then trigger the deployment pipeline, making the comment visible. Sadly it seems that StaticMan no longer works quite as advertised on their site as the bot has been blocked by GitHub for exceeding API rate limits. They still offer a self-hosted version but given I’m running 1GB Linode server memory is quite tight, and I can’t afford a permanently running Node server.

I still liked the idea of StaticMan, so a quick Python script later and I had it integrated with the GitHub API, and Akismet for spam filtering. It runs as a CGI script, so is only spawned when necessary. The script is in the same repository as the rest of my site, but if anyone is interested in reusing I could separate it and release it. Please let me know in the comments below.

Now I’ve completed the update of my blog, it’s time to get writing. I have a few ideas about what I want to talk about, so hopefully it won’t be a couple of years until my next post.

Exposing Docker Ports After The Fact

2017-11-22T12:00:50+00:00

Docker is a great tool for running your applications in a consistent and repeatable environment. One issue that I’ve come across occasionally is getting data into and out of the environment when it’s running.

In this post I want to talk about exposing ports that are published by applications running inside a container. When you start up the container it’s pretty easy to configure the ports you want to expose using the --publish or -p parameter. It’s followed by the internal port number, a colon, and the external port number. For example:

docker run --publish 80:8080 myapp

This will publish port 80 from inside the container as port 8080 on the host.

This works great if you know want ports you want to expose before you run the container. Once it’s running, if you decide you need access to a port, you can’t expose it. Unless that is, you cheat.

socat is a very useful command line tool which lets you create tunnels to forward ports. It has many other features, such as forwarding unix sockets to tcp sockets, but we just need to forward a port from an existing container, into a new container and then expose that port to the host.

Fortunately a Docker container that’s only job is to run socat already exists, so we just need to pass the right options to forward the remote port, and expose the port.

I was trying to expose port 61616 from a container called activemq, so I ran the following command:

docker run -p 61616:61616 alpine/socat tcp-listen:61616,reuseaddr,fork tcp:activemq:61616

Let’s break the command down.

docker run -p 61616:61616

This runs the container and exposes port 61616 on port 61616 on the host.

alpine/socat

This runs the container alpine/socat.

tcp-listen:61616,reuseaddr,fork

This is the first parameter that’s passed to socat. It specifies that it should listen on port 61616.

tcp:activemq:61616

This specifies that when an incoming connection arrives it should be connected port 61616 running on container activemq.

So to summarise, you can run the following command and expose a port while a container is running.

docker run -p cport:hostport alpine/socat tcp-listen:cport,reuseaddr,fork tcp:remotehost:remoteport

Transitioning To A More Open Technology Stack

2017-11-15T12:00:53+00:00

I’m currently working with some large Java monoliths which talk to each other over ActiveMQ. There are several aspects of the architecture that I’d like to change. Certainly, new production environments (Kubernetes, etc) mean that monoliths are not required because of the overhead of deployment, and the benefits of easier testing and more modular architecture mean that I think the expense of migrating to smaller services will be well worth it. With such an established code base though, the question I’m grappling with is how can we transition to a better, more open technology stack without needing to rewrite from scratch and do a big bang deployment.

Currently I’m toying with the idea of writing an ActiveMQ to Web Sockets bridge. Web Sockets are a way of emulating a direct TCP connection in a web browser, although a more normal use case is to send and receive a stream of JSON encoded events. Although Web Sockets were created for use in browsers all languages have libraries available which will allow you to connect to a server.

ActiveMQ natively supports connecting over Web Sockets, so why would I propose building a bridge application? In our case the messages being exchanged are binary encoded, so you can’t decode them unless you’re running Java and have the same library used to send the messages. By building an application to act as a bridge you get much more control over the Web Socket API than if you use the native ActiveMQ implementation, so you can tidy up the JSON representations you use and easily make any other improvements to the API that you want.

Spring is our current Java Framework of choice, which conveniently has a built-in HTTP server which supports Web Sockets. Combining that with our shared library for connecting to ActiveMQ results in a Web Socket server in just a couple of hundred lines of code, and most of that is actually converting the message objects into a nice JSON representation.

In future posts I’ll talk about our progress migrating to a more open environment, but first let’s go through how to build the bridge. I’ve chosen a simple REST API.

GET /topic will return a list of topics.
GET /topic/{topic} returns a single message from the topic (not much use in reality, but useful for testing).
CONNECT /topic/{topic} opens a web socket connection to a topic, which lets you send and receive a stream of events.

The first step is to enable web sockets on the right URL.

@Configuration
@EnableWebSocket
public class WebSocketConfig implements WebSocketConfigurer {
    @Autowired
    private SocketHandler sockerHandler;

    public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
        registry.addHandler(sockerHandler, "/topic/{topic}")
            .setAllowedOrigins("*");
    }
}

Next up we set up the normal HTTP end points. Here I’m using two objects to manage the ActiveMQ connections and JSON serialisation/deserialisation. If like us you have shared libraries to do your messaging for you then you can just plug those in, and there are some many JSON serialisers you can just pick your favourite.

A key thing with this class is to specify the method of the requests so we can use the same URL as we registered for the web sockets without clashing.

@Controller
@RequestMapping("/topic")
public class TopicHandler {
    @Autowired
    private JmsConnectionManager jmsConnectionManager;

    @Autowired
    private JsonSerialiser jsonSerialiser;

    @RequestMapping(method = RequestMethod.GET)
    public @ResponseBody List GetTopics() {
        return jsonSerialiser.serialise(jmsConnectionManager.getTopics());
    }

    @RequestMapping(value="/{topic}", method = RequestMethod.GET, headers = "Connection!=Upgrade")
    public @ResponseBody String GetTopic(@PathVariable("topic") String topic) {
        ActiveMqTopicController controller = jmsConnectionManager.getTopicController(topic);

        return jsonSerialiser.serialise(controller.getMessage(), BaseMessage.class);
    }
}

Lastly, we handle the web socket connections. There are three methods of TextWebSocketHandler that we need to override. handleTextMessage is called when a message is received from the client, while afterConnectionEstablished and afterConnectionClosed are called at the start and end of the connection. When the connection is established you need to connect to the JMS topic, and start streaming events.

@Component
public class SocketHandler extends TextWebSocketHandler {
    @Autowired
    private JmsConnectionManager jmsConnectionManager;

    @Autowired
    private JsonSerialiser jsonSerialiser;

    public SocketHandler() {
    }

    @Override
    public void handleTextMessage(WebSocketSession session, TextMessage message)
            throws InterruptedException {
        BaseMessage jmsMessage = jsonSerialiser.deserialise(message.getPayload(), BaseMessage.class);

        ActiveMqTopicController tc = jmsConnectionManager.getTopicController(getTopic(session));
        tc.publishMessage(jmsMessage);
    }

    @Override
    public void afterConnectionEstablished(WebSocketSession session) throws Exception {
        ActiveMqTopicController tc = jmsConnectionManager.getTopicController(getTopic(session));
        tc.addListener(session);
    }

    @Override
    public void afterConnectionClosed(WebSocketSession session, CloseStatus closeStatus) {
        ActiveMqTopicController tc = jmsConnectionManager.getTopicController(getTopic(session));
        tc.removeListener(session);
    }

    private String getTopic(WebSocketSession session) {
        String path = session.getUri().getRawPath();

        String[] components = path.split("/");

        return components[components.length - 1];
    }
}

With this fairly simple code in place, it’s dead easy to start integrating other languages, or single page apps running in a web browser into your previously closed messaged based system.

Using A Raspberry Pi To Switch On Surround Sound Speakers

2017-11-08T12:00:00+00:00

In a previous post , I talked about network booting a Raspberry Pi MythTV frontend. One issue that I had to solve was how to turn on my Onkyo surround sound speakers, but only if they are not already turned on.

I already had an MCE remote and receiver which can both transmit and receive, so it is perfect for controlling MythTV and switching the speakers on. There are plenty of tutorials out there, but the basic principle is to use irrecord to record the signals from the speaker’s remote control, so the Raspberry Pi can replay them to switch it on when the Pi starts up. In my case, I needed two keys, the power button and VCR/DVR input button. Once you’ve recorded the right signals, you can use irsend to repeat them.

Initially, I had it set up to always send the power button signal on boot. This had the unfortunate side-effect of switching the speakers off if they were already on, for example, if I had been listening to music through Sonos before deciding to watch TV.

To prevent this from happening I needed to determine whether the speakers were on or not. Fortunately, Raspberry Pi’s come with some useful tools to determine information about what is supported by the HDMI device it’s connected to. These tools are tvservice, which dumps the EDID information, and edidparser which turns the EDID into human-readable text.

You can use them as follows:

tvservice -d /tmp/edid.dump

edidparser /tmp/edid.dump > /tmp/edid.txt

This gives you a nice text file containing all of the resolutions and audio formats supported by the connected HDMI device. I took one output when the speakers were on, and one when they were off, and by diffing them I got this set of changes.

    -HDMI:EDID found audio format 2 channels PCM, sample rate: 32|44|48 kHz, sample size: 16|20|24 bits
    +HDMI:EDID found audio format 2 channels PCM, sample rate: 32|44|48|88|96|176|192 kHz, sample size: 16|20|24 bits
    +HDMI:EDID found audio format 6 channels PCM, sample rate: 32|44|48|88|96|176|192 kHz, sample size: 16|20|24 bits
    +HDMI:EDID found audio format 8 channels AC3, sample rate: 32|44|48 kHz, bitrate: 640 kbps
    +HDMI:EDID found audio format 8 channels DTS, sample rate: 44|48 kHz, bitrate: 1536 kbps
    +HDMI:EDID found audio format 6 channels One Bit Audio, sample rate: 44 kHz, codec define: 0
    +HDMI:EDID found audio format 8 channels Dobly Digital+, sample rate: 44|48 kHz, codec define: 0
    +HDMI:EDID found audio format 8 channels DTS-HD, sample rate: 44|48|88|96|176|192 kHz, codec define: 1
    +HDMI:EDID found audio format 8 channels MLP, sample rate: 48|96|192 kHz, codec define: 0

Pretty obvious really - when the speakers are on they support a much greater range of audio formats!

Putting all this together I ended up with the following script. It grabs the EDID data, converts it into text, and if it doesn’t contain DTS-HD then turn the speakers on.

tvservice -d /tmp/edid.dump

edidparser /tmp/edid.dump > /tmp/edid.txt

if ! grep DTS-HD /tmp/edid.txt; then
 irsend SEND_ONCE speaker KEY_POWER
fi

Introducing A New Language

2017-11-01T21:56:35+00:00

At work, there is a discussion going on at the moment about introducing Kotlin into our tech stack. We’re a JVM based team, with the majority of our code written in Java and few apps in Scala. I don’t intend to discuss the pros and cons of any particular language in this post, as I don’t have enough experience of them to decide yet (more on that to come as the discussion evolves). Instead, I wanted to talk about how you can decide when to introduce a new language.

Programmers, myself included, have a habit of being attracted to anything new and shiny. That might be a new library, a new framework or a new language. Whatever it is, the hype will suggest that you can do more, with less code and fewer bugs. The reality often turns out to be a little different, and by the time you have implemented a substantial production system then you’ve probably pushed up against the limits, and found areas where it’s hard to do what you want, or where there are bugs or reliability problems. It’s only natural to look for better tools that can make your life easier.

If you maintain a large, long-lived code base then introducing anything new is something that has to be considered carefully. This is particularly true of a new language. While a new library or framework can have its own learning curve, a new language means the team has to relearn how to do the fundamentals from scratch. A new language brings with it a new set of idioms, styles and best practices. That kind of knowledge is built up by a team over many years, and is very expensive both in time and mistakes to relearn.n Clearly, if you need to start writing code in a radically different environment then you’ll need to pick a new language. If like us, you mostly write Java server applications and you want to start writing modern web-based frontends to your applications then you need to choose to add Javascript, or one of the many Javascript based languages, into your tech stack.

The discussion that we’re having about Java, Scala and Kotlin is nowhere near as clear-cut however. Fundamentally choosing one over the other wouldn’t let us write a new type of app that we couldn’t write before, because they all run in the same environment. Scala is functional, which is a substantial change in idiom, while Kotlin is a more traditional object-orientated language, but considerably more concise than Java.

To help decide it makes sense to write a new application in the potential new language, or perhaps rewrite an existing application. Only with some personal experience can you hope to make a decision that’s not just based on hype, or other people’s experiences. The key is treat this code as a throw-away exercise. If you commit to putting the new app into production, then you’re not investigating the language, you’re commiting to add it to your tech stack before you’ve investigated it.

As well as the technical merits, you should also look into the training requirements for the team. Hopefully there are good online tutorials, or training courses available for your potential technology, but these will need to be collated and shared, and everyone given time to complete them. If you’re switching languages then you can’t afford to leave anyone behind, so training for the entire team is essential.

Whatever you feel is the best language to choose, you need to be bold and decisive in your decision making. If you decide to use a new language for an existing environment then you need to commit to not only writing all new code in it, but to also fairly quickly port all your existing code over as well. Having multiple solutions to the same problem (be it the language you write your server-side, or browser-side apps in, or a library or framework) create massive amounts of duplicated code, duplicated effort and expensive context switching for developers.

Time and again I’ve seen introducing the new shiny solution create a mountain of technical debt because old code is not ported to the new solution, but instead gets left behind in the vague hope that one day it will get updated. New technology and ways of working can have a huge benefit, but never underestimate the cost, and importance, of going all the way.

Network Booting A Raspberry Pi MythTV Frontend

2017-10-25T11:00:37+00:00

When we moved house earlier in the year I wanted to simplify our home theatre setup. With my son starting to grow up, in a normal house he’d be able to turn on the tv and watch his favourite shows without needing us to do it for him, but with the overcomplicated setup that we had it would take him several years longer before he could learn the right sequence of buttons.

I’ve been a MythTV user for well over ten years, and all our TV watching is done through it. At this stage with our history of recorded shows and a carefully curated list of recording rules switching would be a big pain, so I wanted to try and simplify the user experience, even if it means complicating the setup somewhat.

I had previously tried to reduce the standby power consumption by using an Eon Power Down Plug, which monitors the master socket and switches off the slave sockets when the master enters standby mode. This works great as when the TV was off my Xbox and surround speakers would be switched off automatically. The downside is that if I want the use the speakers to listen to music (they are also connected to a Sonos Connect) then either the TV needs to be on, or I need to change the plug over. Lastly, because I was running a combined frontend and backend it wasn’t connected to the smart plug (otherwise it wouldn’t be able to turn on to record.) If you turned the TV off the frontend would still be on, preventing the backend from shutting down for several hours, until it went into idle mode.

I decided to solve these problems by using a Raspberry Pi 3 as a separate frontend, and switching the plugs around. As they run Linux, and have hardware decoding of MPEG2 and h264 they work great as MythTV frontends.

A common issue with Raspberry Pis is that if you don’t shutdown them down correctly then their SD cards become corrupt. If I connected the Pi to the slave plug socket as planned then it would be uncleanly shut down every time the TV was switched off, risking regular corruption. Fortunately Raspberry Pis support network booting, which means you can have the root filesystem mounted from somewhere else, and you don’t even need the SD card at all. I already had a Synology NAS, which I love, and is a perfect host for the filesystem.

Sadly the network code that is built into the Pis ROM (and therefore isn’t updatable) is very specific and buggy. My router’s DNS server doesn’t support the options required to make the Pi boot, so I switched to using a DNS server on the Synology. While you can’t set the right options in the web frontend you can edit the config files directly to make it work. The bugs in the Pis firmware are that the DNS responses must be received at the right time. Too quick or too slow and the Pi will fail to boot. One of the aspects I like the most about my Synology is that it has a very low power suspend more. When it is in this mode it takes a little while to wake up and respond the network event. Waking up takes too long for the Pi, which would give up waiting for a response. While I wouldn’t have been happy about it, I could have disabled the low power mode to make the Pi work. Unfortunately the second time the Pi boots the DNS server responds too quickly (the first time it has to check whether the IP address it is about to hand out is in use.) This response is too quick for the Pi, which again will fail to boot.

The other option is to use an SD card with a kernel and a few supporting files on it to start the boot, and then use Linux’s built-in NFS root filesystem support. While this does require an SD card, it’s read only and after the kernel has been loaded the card will be accessed very rarely, if ever, so the risk of corruption is minimal. After running with this set up for a few months, and being switched off several times per day we’ve not had a single corruption of the SD card so far.

Setting this up is pretty straightforward, I just extracted a Minibian tarball to my NAS and shared it via NFS. Next I copied the contents of /boot to my SD card, and modified cmdline.txt to include the following:

root=/dev/nfs nfsroot=192.168.1.72:/volume1/pi/minibian rw ip=dhcp

With this added it boots up reliably and can be shut down uncleanly with little or no risk of corruption.

Next up is making the MythTV frontend start up automatically. This is was done by adding the following to /etc/rc.local

modprobe rc_rc6_mce
/usr/bin/ir-keytable -c -p RC-5,RC-6 -w /etc/rc_keymaps/rc6_mce
echo "performance" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
su -c "/home/andrew/autostart.sh" andrew &

The first two lines are required to set up my MCE IR receiver. The third line is needed to ensure that the Pi’s performance remains consistent and the CPU isn’t throttled down while you’re in the middle of an episode of Strictly. The final line just triggers another script that actually runs the frontend, but run as me, and not root.

#!/bin/bash

/home/andrew/wake_speakers &
startx /home/andrew/start_myth 2>&1 > ~/mythtv.log

I’ll cover the first line in another post, but it just turns on the surround speakers and makes sure they in the right mode. The second line starts X, and runs my custom start script. This final script looks like this:

#!/bin/bash
QT_QPA_PLATFORM=xcb /usr/bin/mythfrontend -O libCECEnabled=0

While I managed to solve my key issues of making it easier to switch the open and off, and I can listen to music without the TV being on and still have most devices switched fully off, I still have a few issues still to solve. The main two are that bootup speed is not as fast as I would like, and the backend doesn’t cope well with the frontend exiting uncleanly (and it waits 2.5 hours before turning off). I will cover these issues, and some others that I had to solve in a future post.

Links to Amazon contain an affiliate code.

FitBit Ionic Review

2017-10-18T11:00:37+00:00

Since I received my Pebble Steel back in 2014 I knew I never wanted to go back to using a normal watch. Having notifications and apps on my wrist was just too useful to me. I skipped the Pebble Time, but when the Time 2 was announced I happily put in a preorder. Unfortunately it was not to be, and Pebble folded and was sold to FitBit. If Pebble wasn’t able to survive then as an existing FitBit user having them as a buyer is probably the the best option.

The idea of FitBit’s scale and expertise in building hardware, combined with Pebble’s excellent developer platform was an enticing prospect. Rather than switch to an Apple Watch (or Android Wear, although that would have required a new phone) I decide to wait for the fruits of the combined company’s labour to be released.

I was getting a bit itchy, and my trusty Pebble Steel was showing it’s age, but eventually the FitBit Ionic was announced. A few days before the official release date my preorder arrived. It’s now been two weeks of wearing it nearly 24/7, so it seems like a reasonable time to post my thoughts.

First impressions of the hardware are excellent. Most reviews have criticised the looks, but I’m actually a fan. I like the way the bands transition into the watch itself, and sure it does just look like a black square when the screen is off, but that’s the case for all current smart watches. The buttons have a nice firmness to them, and the touchscreen is responsive. I have had some issues swiping to clear notifications, but I think that’s more to do with the touch targets in the software rather than the touchscreen, as I’ve not had issues elsewhere.

The key hardware concerns are the screen and battery life. The bottom line is that both are excellent. The screen is bright and clear, even in strong sunlight. I’ve not tested the battery life extensively because I’m wearing it essentially all day. I only take the Ionic off to shower, and it appears to only lose 15-20% per day, and a quick 15 minute charge per day is enough to keep it topped up.

The one big element I miss from my Pebble is the fact that the screen is not always on. If you do the lift-and-twist “I’m looking at my watch” gesture then it does turn on reliably, but it’s rare that I actually do that. Looking at my watch tends to be a much more subtle movement, and then it only recognises it occasionally. I have found myself pressing a button to turn the screen on, which after having an always on screen feels like a step backwards.

At the moment it’s probably too early to comment on the software side. The core features are all there and work well. Notifications from apps, texts and calls all work. I’ve been able to track various types of exercise, including bike rides which were tracked with the built in GPS and synced automatically to Strava. Heart rate monitoring and step count also appear reasonably accurate, as you would expect given FitBit’s history.

Unfortunately the key reason I brought the Ionic - that they had Pebble’s software team building the SDK - is not yet visible. There are a small set of watch faces (I’m a fan of the Cinemagraph), and some built in apps, but as yet there’s no sign of any externally developed apps. It’s early days though, and hopefully a developer community will form soon.

So, would I recommend the FitBit Ionic? Yes, but more on potential than current execution. The hardware appears to be there, it just needs a bit more time for the software to mature and apps to be developed. FitBit Ionic photograph by FitBit.

Leading Without Deep Technical Knowledge

2017-10-11T12:20:55+00:00

In my previous jobs, when I’ve been promoted to a leadership role it has been as a result of being the most experienced member on the team. Having a deep knowledge of the business, the code base and the technologies we’re using meant I was already an authority on most topics of the team needed to discuss, and could weigh in on a discussion with a well formed and considered option.

When I changed companies at the end of last year I came to Ocado Technology as a team lead for an existing team, using a technology stack I wasn’t familiar with. In fact Ocado are a Java based company, which I had never used before, so not only was I not familiar with the frameworks and libraries used, but I wasn’t even familiar with language the code was written it either!

Leading in a situation like this required a complete change in how I approached problems. When a stakeholder or the product owner approached me with a challenge rather than immediately being able to respond with a rough solution, and vague estimate or a timeline I need to defer to my team, and let them propose a solution, estimate it, and then I could fit it into our schedule. I might challenge them on some points, but it was their plan. I quickly needed to learn who knew the most about which systems, so I could get the right people involved in discussions early.

Previously although I was able give initial feedback on a potential project, I would still allow the team to discuss them, to propose alternate solutions and to estimate. The change is that now my contribution is much more about making sure the right people are talking and helping to avoid misunderstanding when the business and my developers are accidentally talking at cross-purposes.

While this change has definitely pushed me out of my comfort zone, it has also given me space to focus a different area of my leadership skills. Ocado prides itself on its values, one of which is its servant leadership philosophy. By not having the knowledge to make decisions myself I am forced to empower my team to make decisions on how they want solve problems.

It’s not just case a facilitating discussions though. I may not know the details of our code base, or the intricacies of library, but my knowledge of software design patterns and systems architecture is valid whatever language is being used, and my opinions are as strong as ever. It is normal for developers to immediately jump to the simplest solution to a problem within the framework of the existing code. As an outsider my first instinct is usually to take a step back and ask why the system is designed like that, and to propose a bigger solution that resolves some technical debt, rather than focussing on the issue at hand.

This change in role has made me realise that even when I was the most experienced in the code, language or framework I should have made more of an effort to devolve the decision making process. Not to stop expressing my opinions, or involving myself in the discussions, but to explicitly encourage others to contribute, and make sure they are taking part in discussions. This has resulted in people being more bought in to solutions, and encouraged a much closer team with a greater feeling of ownership over our code. Being forced to make this change to my style has undoubtedly made me a better manager, and a better developer too.

Accessing FitBit Intraday Data

2012-12-30T13:22:03+00:00

For Christmas my wife and I brought each other a new FitBit One device (Amazon affiliate link included). These are small fitness tracking devices that monitor the number of steps you take, how high you climb and how well you sleep. They’re great for providing motivation to walk that extra bit further, or to take the stairs rather than the lift.

I’ve only had the device for less than a week, but already I’m feeling the benefit of the gamification on FitBit.com. As well as monitoring your fitness it also provides you with goals, achievements and competitions against your friends. The big advantage of the FitBit One over the previous models is that it syncs to recent iPhones, iPads, as well as some Android phones. This means that your computer doesn’t need to be on, and often it will sync without you having to do anything. In the worst case you just have to open the FitBit app to update your stats on the website. Battery life seems good, at about a week.

The FitBit apps sync your data directly to FitBit.com, which is great for seeing your progress quickly. They also provide an API for developers to provide interesting ways to process the data captured by the FitBit device. One glaring omission from the API is any way to get access to the minute by minute data. For a fee of $50 per year you can become a Premium member which allows you do to a CSV export of the raw data. Holding the data, collected by a user hostage is deeply suspect and FitBit should be ashamed of themselves for making this a paid for feature. I have no problem with the rest of the features in the Premium subscription being paid for, but your own raw data should be freely available.

The FitBit API does have the ability to give you the intraday data, but this is not part of the open API and instead is part of the ‘Partner API’. This does not require payment, but you do need to explain to FitBit why you need access to this API call and what you intend to do with it. I do not believe that they would give you access if your goal was to provide a free alternative to the Premium export function.

So, has the free software community provided a solution? A quick search revealed that the GitHub user Wadey had created a library that uses the urls used by the graphs on the FitBit website to extract the intraday data. Unfortunately the library hadn’t been updated in the last three years and a change to the FitBit website had broken it.

Fortunately the changes required to make it work are relatively straightforward, so a fixed version of the library is now available as andrewjw/python-fitbit. The old version of the library relied on you logging into to FitBit.com and extracting some values from the cookies. Instead I take your email address and password and fake a request to the log in page. This captures all of the cookies that are set, and will only break if the log in form elements change.

Another change I made was to extend the example dump.py script. The previous version just dumped the previous day’s values, which is not useful if you want to extract your entire history. In my new version it exports data for every day that you’ve been using your FitBit. It also incrementally updates your data dump if you run it irregularly.

If you’re using Windows you’ll need both Python and Git installed. Once you’ve done that check out my repository at github.com/andrewjw/python-fitbit. Lastly, in the newly checked out directory run python examples/dump.py <email> <password> <dump directory>.

Losing Games

2012-05-29T19:24:45+00:00

I’m not a quick game player. I don’t rush out a buy the latest games and complete them on the same weekend. Currently I’m most of the way through both Alan Wake and L.A. Noire.

Alan Wake is a survival horror game where you’re fighting off hordes of people possessed by darkness. L.A. Noire is a detective story that has you solving crimes in 1940s Los Angeles. Both feature an over the shoulder third person camera, and both have excellent graphics. They also both have a film like quality to the story. In Alan Wake the action is divided up in six tv style “episodes”, with a title sequence between each one. It also has a number of cut scenes and narration by the title character sprinkled throughout the game which help to drive the story forward.

In L.A. Noire you are detective try to solve crimes and rise up the ranks of the police force. The game features cut scenes to introduce and close each case. During each case you head from location to location and interviewing suspects and witnesses. The big breakthrough in L.A. Noire is the facial animation in the game. Rather than being animated by hand the faces of characters were recorded directly from actor’s faces. This gives the faces a lifelike quality that has not been seen in games before.

Despite the extensive similarities between the game my opinion of the two could hardly be more different. Alan Wake is one of the best games I’ve ever played, while L.A. Noire is really quite boring. I was trying to work out why I felt so differently about them when I read the following quote in Making Isometric Social Real-Time Games with HTML5, CSS3, and JavaScript by Mario Andres Pagella.

This recent surge in isometric real-time games was caused partly by Zynga’s incredible ability to “keep the positive things and get rid of the negative things” in this particular genre of games, and partly by a shift in consumer interests. They took away the frustration of figuring out why no one was “moving to your city” (in the case of SimCity) and replaced it with adding friends to be your growing neighbours.

The need for the face of characters in L. A. Noire to be recorded from real actors limits one of the best things about games: their dynamic nature. Even if you get every question wrong you still solve the case and make progress. Initially you don’t really notice this, but quickly I found it meant that the questioning, the key game mechanic, became superfluous.

Alan Wake is a fairly standard game in that there’s really only one way to progress. This is well disguised though so you don’t notice. The atmosphere in the game forces you to keep moving and the story progresses at quite a pace.

Ultimately it’s not for me to criticise what games people want to play. FarmVille and the rest of Zynga’s games are enormously popular. What disappoints me most about L.A. Noire is that it’s such a technically advanced game, but falls down on such a simple piece of game mechanics. Alan Wake on the other hand succeeds mostly based on story and atmosphere, and that’s the way it should be.

Scalable Collaborative Filtering With MongoDB

2012-03-28T12:46:04+00:00

Many websites have some form of recommendation system. While it’s simple to create a recommendation system for small amounts of data, how do you create a system that scales to huge amounts of data?

How to actually calculate the similarity of two items is a complicated topic with many possible solutions. Which one if appropriate depends on your particularly application. If you want to find out more I suggest reading the excellent Programming Collective Intelligence (Amazon affiliate link) by Toby Segaran.

We’ll take the simplest method for calculating similarity and just calculate the percentage of users who have visited both pages compared to the total number who have visited either. If we have Page 1 that was visited by user A, B and C and Page 2 that was visited by A, C and D then the A and C visited both, but A, B, C and D visited either one so the similarity is 50%.

With thousands or millions of items and millions or billions of views calculating the similarity between items becomes a difficult problem. Fortunately MongoDB’s sharding and replication allow us to scale the calculations to cope with these large datasets.

First let’s create a set of views across a number of items. A view is stored as a single document in MongoDB. You would probably want to include extra information such as the time of the view, but for our purposes this is all that is required.

views = [
        { "user": "0", "item": "0" },
        { "user": "1", "item": "0" },
        { "user": "1", "item": "0" },
        { "user": "1", "item": "1" },
        { "user": "2", "item": "0" },
        { "user": "2", "item": "1" },
        { "user": "2", "item": "1" },
        { "user": "3", "item": "1" },
        { "user": "3", "item": "2" },
        { "user": "4", "item": "2" },
    ]
for view in views:
    db.views.insert(view)

The first step is to process this list of view of events so we can take a single item and get a list of all the users that have viewed it. To make sure this scales over a large number of views we’ll use MongoDB’s map/reduce functionality.

def article_user_view_count():
    map_func = """
function () {
    var view = {}
    view[this.user] = 1
    emit(this.item, view);
}
"""

We’ll build a javascript Object where the keys are the user id and the value is the number of time that user has viewed this item. In the map function we we build an object that represents a single view and emit it using the item id as the key. MongoDB will group all the objects emitted with the same key and run the reduce function, shown below.

    reduce_func = """
function (key, values) {
    var view = values[0];
    for (var i = 1; i < values.length; i++) {
        for(var item in values[i]) {
            if(!view.hasOwnProperty(item)) { view[item] = 0; }
            view[item] = view[item] + values[i][item];
        }
    }
    return view;
}
"""

A reduce function takes two parameters, the key and a list of values. The values that are passed in can either be those emitted by the map function, or values returned from the reduce function. To help it scale not all of the original values will be processed at once, and the reduce function must be able to handle input from the map function or its own output. Here we output a value in the same format as the input so we don’t need to do anything special.

    db.views.map_reduce(Code(map_func), Code(reduce_func), out="item_user_view_count")

The final step is to run the functions we’ve just created and output the data into a new collection. Here we’re recalculating all the data each time this function is run. To scale properly you should filter the input based on the date the view occurred and merge it with the output collection, rather than replacing it as we are doing here.

Now we need calculate a matrix of similarity values, linking each item with every other item. First lets see how we can calculate the similarity of all items to one single item. Again we’ll use map/reduce to help spread the load of running this calculation. Here we’ll just use the map part of map/reduce because each input document will be represented by a single output document.

def similarity(item):
    map_func = """
function () {
    if(this._id == "%s") { return; }
    var viewed_both = {};
    var viewed_any = %s;
    for (var user in this.views) {
        if(this.value.hasOwnProperty(user)) {
            viewed_both[user] = 1;
        }n
        viewed_any[user] = 1;
     }n
     emit("%s"+"_"+this._id, viewed_both.length / viewed_any.length );
}
""" % (int(item["_id"]), json.dumps(item["value"]), json.dumps(item["value"]) int(item["_id"]), )

The input to our Python function is a document that was outputted by our previous map/reduce call. We build a new Javascript by interpolating some data from this document into a template function. We loop through all the users who viewed the document we’re comparing against and work out whether they have viewed both. At the end of the function we emit the percentage of users who viewed both.

    reduce_func = """
function (key, values) {
    return results[0];
}
"""

Because we output unique ids in the map function this reduce function will only be called with a single value so we just return that.

    db.item_user_view_count.map_reduce(Code(map_func), Code(reduce_func), out=SON([("merge", "item_similarity")]))

The last step in this function is to run the map reduce. Here as we’re running the map/reduce multiple times we need to merge the output rather than replacing it as we did before.

The final step is to loop through the output from our first map/reduce and call our second function for each item.

for doc in db.item_user_view_count.find():
    similarity(doc)

A key thing to realise is that you don’t need to calculate live similarity data. Once you have even a few hundred views per item then the similarity will remain fairly consistent. In this example we step through each item in turn and calculate the similarity for it with every other item. For a million item database where each iteration of this loop takes one second the similarity data will be updated once every 11 days.

I’m not claiming that you can take the code provided here and immediately have a massively scalable system. MongoDB provides an easy to use replication and sharding system, which are plugged in to its Map/Reduce framework. What you should take away is that by using map/reduce with sharding and replication to calculate the similarity between two items we can quickly get a system that scales well with an increasing number of items and of views.

Steve Jobs and the Lean Startup

2012-03-20T13:50:36+00:00

On my 25 minute train journey to work each morning I like to pass the time by reading. The two most recent books I’ve read are The Lean Startup: How Constant Innovation Creates Radically Successful Businesses by Eric Ries and Steve Jobs by Walter Isaacson (both links contain an affiliate id). Although one is a biography and the other is a book on project management they actually cover similar ground, and both are books that people working in technology should read.

Walter Isaacson’s book has been extensively reviewed and dissected so I’m not going to go into detail on it. The book is roughly divided into two halves. The first section is on the founding of Apple, Pixar and NeXT. This section serves an inspirational guide to setting up your own company. The joy of building a great product and defying the odds against a company succeeding comes across very strongly. The later section following Job’s return to Apple is a much more about the nuts and bolts of running a huge corporation. While it’s an interesting guide to how Apple got to where it is today, it lacks the excitement of the earlier chapters.

The Lean Startup could, rather unkindly, be described as a managerial technique book. It’s much more than that though as it’s more of a philosophy for how to run company or a project. The book is very readable and engaging with plenty of useful case studies to illustrate the point being made. The key message of the book is to get your product out to customers as soon as possible, to measure as much as you can and learn from what your customers are doing and saying. As you learn you need to make a decision on whether to persevere or to pivot, and change strategy.

There are many reasons why Steve Jobs was a great leader, a visionary and a terrible boss. One aspect was his unshakable belief that he knew what the customer wanted, even before they knew themselves. This is the antithesis of the Lean Startup methodology, which focuses on measurement and learning. Eric Ries stresses that a startup is not necessarily two guys working out of a garage. Huge multinational corporations can have speculative teams or projects inside them, that act much like start ups, so it wouldn’t be impossible for the Apple of today to act like a start up. Apple weren’t always huge though, and back in the 1970s they really were a start up.

One Apple trait the Lean Startup methodolgy doesn’t allow for is dramatic product launches. The Lean Startup is a way of working that relies on quick iteration and gradually building up your customer base. It’s hard to quickly iterate when building hardware, but early in Apple’s life they were struggling to find a market for their computers. The Apple I follow the trend of the time of build-it-yourself computers. Just a year later and Apple released the [Apple ] which came with a case and was much more suitable for the average consumer. This represents a pivot on the part of Apple. They could have continued to focus on hobbyists but instead they decided to change and aim for a bigger, but less technical, market.

Reading is a key part of becoming a better programmer. Whether it’s reading about the latest technology on a blog, the latest project management techniques or the history of computers reading will help you become better at your job. I’m not sure I recommend anyone tries to recreate Steve Job’s management style, but as a history of Apple Walter Isaacson’s book is inspirational and informative. The Lean Startup is considerably more practical, even if it won’t inspire you to set a company in the first place.

Django ImportError Hiding

2012-03-07T13:59:29+00:00

A little while ago I was asked what my biggest gripe with Django was. At the time I couldn’t think of a good answer because since I started using Django in the pre-1.0 days most of the rough edges have been smoothed. Yesterday though, I encountered an error that made me wish I thought of it at the time.

The code that produced the error looked like this:

from django.db import models
class MyModel(model.Model):
    ...
    def save(self):
        models.Model.save(self)
        ...
    ...

The error that was raised was AttributeError: 'NoneType' object has no attribute 'Model'. This means that rather than containing a module object, models was None. Clearly this is impossible as the class could not have been created if that was the case. Impossible or not, it was clearly happening.

Adding a print statement to the module showed that when it was imported the models variable did contain the expected module object. What that also showed was that module was being imported more than once, something that should also be impossible.

After a wild goose chase investigating reasons why the module might be imported twice I tracked it down to the load_app method in django/db/models/loading.py. The code there looks something like this:

    def load_app(self, app_name, can_postpone=False):
        try:
            models = import_module('.models', app_name)
        except ImportError:
            # Ignore exception

Now I’m being a harsh here, and the exception handler does contain a comment about working out if it should reraise the exception. The issue here is that it wasn’t raising the exception, and it’s really not clear why. It turns out that I had a misspelt module name in an import statement in a different module. This raised an ImportError which was caught, hidden and then Django repeatedly attempted to import the models as they were referenced in the models of other apps. The strange exception that was originally encountered is probably an artefact of Python’s garbage collection, although how exactly it occurred is still not clear to me.

There are a number of tickets (#6379, #14130 and probably others) on this topic. A common refrain in Python is that it’s easier to ask for forgiveness than to ask for permission, and I certainly agree with Django and follow that most of the time.

I always follow the rule that try/except clauses should cover as little code as possible. Consider the following piece of code.

try:
    var.method1()n
    var.member.method2()
except AttributeError:
    # handle error

Which of the three attribute accesses are we actually trying to catch here? Handling exceptions like this are a useful way of implementing Duck Typing while following the easier to ask forgiveness principle. What this code doesn’t make clear is which member or method is actually optional. A better way to write this would be:

var.method1()
try:
    member = var.member
except AttributeError:
    # handle error
else:
    member.method2()

Now the code is very clear that the var variable may or may not have a member member variable. If method1 or method2 do not exist then the exception is not masked and is passed on. Now lets consider that we want to allow the method1 attribute to be optional.

try:
    var.method1()
except AttributeError:
    # handle error

At first glance it’s obvious that method1 is optional, but actually we’re catching too much here. If there is a bug in method1 that causes an AttributeError to raised then this will be masked and the code will treat it as if method1 didn’t exist. A better piece of code would be:

try:
    method = var.method1
except AttributeError:
    # handle error
else:
    method()

ImportErrors are similar because code can be executed, but then when an error occurs you can’t tell whether the original import failed or whether an import inside that failed. Unlike with an AttributeError there is a no easy way to rewrite the code to only catch the error you’re interested in. Python does provide some tools to divide the import process into steps, so you can tell whether the module exists before attempting to import it. In particular the ` imp.find_module` function would be useful.

Changing Django to avoid catching the wrong ImportErrors will greatly complicate the code. It would also introduce the danger that the algorithm used would not match the one used by Python. So, what’s the moral of this story? Never catch more exceptions than you intended to, and if you get some really odd errors in your Django site watch out for ImportErrors.

Back Garden Weather in CouchDB (Part 5)

2012-02-10T20:00:07+00:00

After a two week gap the recent snow in the UK has inspired me to get back to my series of posts on my weather station website, WelwynWeather.co.uk. In this post I’ll discuss the records page, which shows details such as the highest and lowest temperatures, and the heaviest periods of rain.

From a previous post in this series you’ll remember that the website is implemented as a CouchApp. These are Javascript functions that run inside the CouchDB database, and while they provide quite a lot of flexibility you do need to tailor your code to them.

On previous pages we have use CouchDB’s map/reduce framework to summarise data then used a list function to display the results. The records page could take a similar approach, but there are some drawbacks to that. Unlike the rest of the pages the information on the records page consists of a number of unrelated numbers. While we could create a single map/reduce function to process all of them at once. That function will quickly grow and become unmanageable, so instead we’ll calculate the statistics individually and use AJAX to load them dynamically into the page.

To calculate the minimum indoor temperature we first need to create a simple view to calculate the value. As with all CouchDB views this starts with map function that outputs the parts of the document we are interested in.

function(doc) {
    emit(doc._id, { "temp_in": doc.temp_in, "timestamp": doc.timestamp });
}

Next we create a reduce function to find the lowest temperature. To do this we simply loop through all the values and select the smallest temperature, recording the timestamp that temperature occurred.

function(keys, values, rereduce) {
    var min = values[0].temp_in;
    var min_on = values[0].timestamp;n
    for(var i=0; i<values.length; i++) {
        if(values[i].temp_in < min) {
            min = values[i].temp_in;
            min_on = values[i].timestamp;
        }
    }n
    return { "temp_in": min, "timestamp": min_on }
}

The website welwynweather.co.uk actually points to the Couch rewrite document. To make the view available we add a rewrite to expose it to the world. As we want to reduce all documents to a single point we just need to pass reduce=true as the query.

{
    "from": "/records/temperature/in/min",
    "to": "/_view/records_temp_in_min",
    "query": { "reduce": "true" }
},

Lastly we can use jQuery to load the data and place the values into the DOM at the appropriate place. As CouchDB automatically sends the correct mime type jQuery will automatically decode the JSON data making this function very straightforward.

$.getJSON("records/temperature/in/min", function (data, textStatus, jqXHR) {
    var row = data.rows[0].value;
    var date = new Date(row.timestamp*1000);
    $("#min_temp_in").html(row.temp_in);
    $("#min_temp_in_date").html(date.toUTCString());
  });

This approach works well for most of the records that I want to calculate. Where it falls down is when calculating the wettest days and heaviest rain as the data needs to be aggregated before being reduced to a single value. Unfortunately CouchDB does not support this. The issue is that you cannot guarantee that the original documents will be passed to your view in order. In fact it is more likely than not than they won’t be. So, to calculate the heaviest periods of rain you would need to build a data structure containing each hour or day and the amount of rain in that period. As the documents are processed the structure would need to be updated and the period with the highest rain found.

Calculating a complicated structure as the result of your reduce function is disallowed by CouchDB, for good reason. An alternative way to find the heaviest periods of rain would be to put the output of the aggregation function into a new database and run another map/reduce function over that to find the heaviest period. Unfortunately CouchDB doesn’t support the chaining of views, so this is impossible without using an external program.

To solve this problem I do the aggregation in CouchDB and the transfer the whole result to the webbrowser and calculate the heaviest period in Javascript. The code to do this is given below. It’s very similar to that given above, but includes a loop to cycle over the results and pick the largest value.

$.getJSON("records/rain/wettest", function (data, textStatus, jqXHR) {
        var max_on = data.rows[0].key;
        var max_rain = data.rows[0].value;
        for(var i=0; i<data.rows.length; i++) {
            if(data.rows[i].value > max_rain) {
                max_on = data.rows[i].key;
                max_rain = data.rows[i].value;
            }
        }
        var date = new Date(max_on*1000);
    $("#wettest_day").html(max_rain);
        $("#wettest_day_date").html(date.toDateString());
    });

This solution works ok, but as time goes on the dataset gets bigger and bigger and the amount of data that is transferred to the browser will grow and grow. Hopefully in future I’ll be able to write another post about changing this to use chained viewed.

CouchDB is a great document store that is at home the web. The ability to run simple sites right from your database is extremely useful and makes deployment a snap. As with all technology you need to be aware of the limitations of CouchDB and allow for them in your designs. In my case the inability to chain views together is really the only wart in the code. Don’t forget you can replicate the database to get the data and use the couchapp command to clone a copy of site. See the first post in this series for instructions on how to do this. Please let me know in the comment section below if you find the site useful or have any questions or comments on the code.

Hackathons, and why your company needs one

2012-02-02T14:08:58+00:00

I could wax lyrical about how programming is an art form and requires a great deal of creativity. However, it’s easy to loose focus on this in the middle of creating project specs and servicing your technical debt. Like many companies we recently held a hackathon event where we split up into teams and worked on projects suggested by the team members.

Different teams took different approaches to the challenge, one team set about integrating an open source code review site in our development environment, others investigated how some commercial technologies could be useful to us. My team built a collaborative filtering system using MongoDB. I’ll post about that project in the future, but in this post I wanted to focus on what we learnt about running a company Hackathon event.

If you’re lucky you’ll work in a company that’s focused on technology and you’ll always be creating new and interesting things. In the majority of companies technology is a means to a end, rather than the goal. In that case it’s easy to become so engrossed in the day to day work that you forget to innovate or to experiment with new technologies. A hackathon is a great way to take a step back and try something new for a few days.n Running a hackathon event should be divided into three stages, preparation, the event and the post event. Before the event you need to take some time to collect ideas and do some preliminary research. The event itself should be a whirlwind of pumping out code and building something exciting. Afterwards you need to take some time to demonstrate what you’ve built, and share what you’ve learnt.

Typical IT departments will been given a set of requirements and will need work out how to achieve them. What a hackathon event should allow is for the department to produce their own set of requirements, free from any external influences. In your day to day work what projects you actually tackle will be decided by a range of factors, but a hackathon is designed to let programmers take something where they’ve thought “we could do something interesting with that data” or “that technology might help us make things better” and run with it. The first stage is to collect all these ideas from the department and then to divide the team up into groups to tackle the most popular projects, perhaps with a voting staging to whittle the ideas down. To keep things fun and quick small teams are key here, any more than five people and you’ll start to get bogged down in process and design by committee.

Once you’ve got the projects and teams sorted you can prepare for the event. People who are working on each project need to think about what they want to have built by the end of event and should be dreaming up ways to tackle the problem. Coding before is banned, but things will go much quicker if you’ve come up with a plan to attack the problem.

For the event you need to remove as many distractions as possible. Certainly you need tell others in the company that you will not be available for a few days. Whether other possibilities such as not reading your email are doable depends on how often you need to deal with crises. Hopefully with no-one fiddling with your servers fewer things will go wrong than on an average day. Moving location, either to meeting rooms or to an external space are both good ways of getting space to focus on the work.

Once the time has expired you need to wrap the event up, and the key thing is to demonstrate what you’ve built to the rest of team. Not everyone in IT is happy with standing and presenting, but a few minutes to people they know well should not be a problem. It’s tempting to invite people from outside the department to the presentations, and before my first hackathon I was very keen on bringing management, who often won’t understand what IT do, into the meeting. Hopefully what is built will show how innovative and full of ideas programmers can be. In reality though the best you can really hope for is a set of tech demos that require quite a lot of understanding about the limitations inherent in building things quickly, which those outside IT will struggle to understand.

The presentations should focus on two things, a demonstration of what you’ve built and a discussion on the technology used and decisions made. The aim should be to excite the team about what you’ve built and to impart some of what you’ve learnt. A good way to spark a discussion is to talk about what problems you encountered. How long the presentations should be depends a lot on the complexity of the problems being tackled, but ten to twenty minutes per project is a good length of time.

In the longer term after the event you’ll need to decide which of the projects to keep, which to improve on or rewrite and which to abandon completely. In a more structured and prepared presentation showing the projects to management maybe a good idea. The presentations immediately following the hackathon are likely to be a fairly ramshackle affair and won’t make the most ‘professional’ impression.

Traditional ‘training’ session don’t work to well in IT, it’s a easier to learn by doing. Most people are also quite sceptical of ‘build a raft’ style team building exercises, compared to those hackathons are the perfect mix of learning and fun. They’re also a great way to work on the problems that you’ve wanted to solve for ages but have never found the time. Even if you don’t get anything you can use out of the event the process of getting there will be worthwhile. And who knows, you might build a million dollar product for your company.

Integrating Python and Javascript with PyV8

2012-01-23T14:15:51+00:00

A hobby project of mine would be made much easier if I could run the same code on the server as I run in the web browser. Projects like Node.js have made Javascript on the server a more realistic prospect, but I don’t want to give up on Python and Django, my preferred web development tools.

The obvious solution to this problem is to embed Javascript in Python and to call the key bits of Javascript code from Python. There are two major Javascript interpreters, Mozilla’s SpiderMonkey and Google’s V8.

Unfortunately the python-spidermonkey project is dead and there’s no way of telling if it works with later version of SpiderMonkey. The PyV8 project by contrast is still undergoing active development.

Although PyV8 has a wiki page entitled How To Build it’s not simple to get the project built. They recommend using prebuilt packages, but there are none for recent version of Ubuntu. In this post I’ll describe how to build it on Ubuntu 11.11 and give a simple example of it in action.

The first step is make sure you have the appropriate packages. There may be others that are required and not part of the default install, but there are what I had to install.

sudo aptitude install scons libboost-python-dev

Next you need to checkout both the V8 and PyV8 projects using the commands below.

svn checkout http://v8.googlecode.com/svn/trunk/ v8
svn checkout http://pyv8.googlecode.com/svn/trunk/ pyv8

The key step before building PyV8 is to set the V8_HOME environment variable to the directory where you checked out the V8 code. This allows PyV8 to patch V8 and build it as a static library rather than the default dynamic library. Once you’ve set that you can use the standard Python setup.py commands to build and install the library.

cd v8
export PyV8=`pwd`
cd ../pyv8
python setup.py build
sudo python setup.py install

In future I’ll write more detailed posts about how to use PyV8, but let’s start with a simple example. Mustache is a simple template language that is ideal when you want to create templates in Javascript. There’s actually a Python implementation of Mustache, but let’s pretend that it doesn’t exist.

To start import the PyV8 library and create a JSContext object. These are equivalent to sub-interpreters so you have several instance of your Javascript code running at once.

>>> import PyV8
>>> ctxt = PyV8.JSContext()

Before you can run any Javascript code you need enter() the context. You should also exit() it when you are complete. JSContext objects can be used with with statements to automate this, but for a console session it’s simplest to call the method explicitly. Next we call eval() to run our Javascript code, first by reading in the Mustache library and then to set up our template as a variable.

>>> ctxt.enter()
>>> ctxt.eval(open("mustache.js").read())
>>> ctxt.eval("var template = 'Javascript in Python is ';")

The final stage is to render the template by dynamically created some Javascript code. The results of the expressions are returned as Python objects, so here rendered contains a Python string.

>>> import random
>>> opinion = random.choice(["cool", "great", "nice", "insane"])
>>> rendered = ctxt.eval("Mustache.to_html(template, { opinion: '%s' })" % (opinion, ))
>>> print rendered
Javascript in Python is nice

There’s much more to PyV8 than I’ve described in this post, including calling Python code from Javascript but unfortunately the V8 and PyV8 documentation is a bit lacking. I will post some more of my discoveries in future posts.

Back Garden Weather in CouchDB (Part 4)

2012-01-20T14:15:19+00:00

In this series of posts I’m describing how I created a CouchDB CouchApp to display the weather data collected by the weather station in my back garden. In the previous post I showed you how to display a single day’s weather data. In this post we will look at processing the data to display it by month.

The data my weather station collects consists of a record every five minutes. This means that a 31 day month will consist of 8,928 records. Unless you have space to draw a graph almost nine thousand pixels wide then there is no point in wasting valuable rending time processing that much data. Reducing the data to one point per hour gives us a much more manageable 744 data points for a month. A full years worth of weather data consists of 105,120 records, even reducing it to one point per hour gives us 8760 points. When rendering a year’s worth of data it is clearly worth reducing the data even further, this time to one point per day.

How do we use CouchDB to reduce the data to one point per hour? Fortunately CouchDB’s map/reduce architecture is perfect for this type of processing. CouchDB will also cache the results of the processing automatically so it only needs to be run once rather than requiring an expensive denormalisation process each time some new data is uploaded.

First we need to group the five minute weather records together into groups for each hour. We could do this by taking the unix timestamp of record and rounding to the nearest hour. The problem with this approach is that the keys are included in the urls. If you can calculate unix timestamps in your head then your maths is better than mine! To make the urls more friendly we’ll use a Javascript implementation of sprintf to build a human-friendly representation of date and time, excluding the minute component.

function(doc) {
    // !code vendor/sprintf-0.6.jsn
    emit(sprintf("%4d-%02d-%02d %02d", doc.year, doc.month, doc.day, doc.hour), doc);
}

CouchDB will helpfully group documents with the same key, so all the records from the same hour will be passed to the reduce function. What you cannot guarantee though is that all the records will be passed in one go, instead you must ensure that your reduce function can operate on its own output. You can tell whether you are ‘rereducing’ the output of the reduce function by checking the third parameter to the function.

function(keys, values, rereduce) {
    var count = 0;n
    var timestamp = values[0].timestamp;
    var temp_in = 0;
    var temp_out = 0;
    var abs_pressure = 0;
    var rain = 0;n
    var wind_dir = [];
    for(var i=0; i<8; i++) { wind_dir.push({ value: 0}); }

To combine the multiple records it makes sense to average most of the values. The exceptions to this are the amount of rain, which should be summed; the wind direction, which should be a count of the gusts in each direction, and the wind gust speed which should be the maximum value. Because your reduced function may be called more than once calculating the average value is not straightforward. If you simply calculate the average of the values passed in then you will be calculating the average of averages, which is not the same the average of the full original data. To work around this we calculate the average of the values and store that with the number of values. Then, when we rereduce, we multiply the average by the number of values and then average the multiplied value.

In the previous, simplified, code snippet we set up the variables that will hold the averages.n

    for(var i=0; i<values.length; i++) {
        var vcount;
        if(rereduce) { vcount = values[i].count } else { vcount = 1 }

We now loop through each of the values and work out how many weather records the value we’re processing represents. The initial pass will just represent a single record, but in the rereduce step it will be more.

        temp_in = temp_in + values[i].temp_in * vcount;
        temp_out = temp_out + values[i].temp_out * vcount;
        abs_pressure = abs_pressure + values[i].abs_pressure * vcount;

Here we build up the total values for temperature and pressure. Later we’ll divide these by the number of records to get the average. The next section adds the rain count up and selects the maximum wind gust.

        rain = rain + values[i].rain;
        wind_ave = wind_ave + values[i].wind_ave * vcount;
        if(values[i].wind_gust > wind_gust) { wind_gust = values[i].wind_gust; }

So far we’ve not really had to worry about the possibility of a rereduce, but for wind direction we need to take it into account. An individual record has a single window direction but for a hourly records we want to store the count of the number of times each direction was recorded. If we’re rereducing we need to loop through all the directions and combine them.

        if(rereduce) {
            for(var j=0; j<8; j++) {
                wind_dir[j]["value"] += values[i].wind_dir[j]["value"];
            }
        } else if(values[i].wind_ave > 0 && values[i].wind_dir >= 0 && values[i].wind_dir < 16) {
            wind_dir[Math.floor(values[i].wind_dir/2)]["value"] += 1;
        }
        if(values[i].timestamp < timestamp) { timestamp = values[i].timestamp; }
        count = count + vcount;
    }

The final stage is to build the object that we’re going to return. This stage is very straightforward, we just need to divide the numbers we calculated before by the count of the number of records. This gives us the correct average for these values.

    return {
            "count": count,
            "status": status,
            "timestamp": timestamp,
            "temp_in": temp_in / count,
            "temp_out": temp_out / count,
            "abs_pressure": abs_pressure / count,
            "rain": rain,
            "wind_ave": wind_ave / count,
            "wind_gust": wind_gust,
            "wind_dir": wind_dir,
        };
}

Now we have averaged the weather data into hourly chunks we can use a list, like the one described in the previous post, to display the data.

In the next and final post in this series I’ll discuss the records page on the weather site.

Back Garden Weather in CouchDB (Part 3)

2012-01-12T13:46:17+00:00

In this series I’m describing how I used a CouchDB CouchApp to display the weather data collected by a weather station in my back garden. In the first post I described CouchApps and how to get a copy of the site. In the next post we looked at how to import the data collected by PyWWS and how to render a basic page in a CouchApp. In the post we’ll extend the basic page to display real weather data.

Each document in the database is a record of the weather data at a particular point in time. As we want to display the data over a whole day we need to use a list function. list functions work similarly to the show function we saw in the previous post. Unlike show functions list functions don’t have the document passed in, they can call a getRow function which returns the next row to process. When there are no rows left it returns null.

show functions process an individual document and return a single object containing the processed data and any HTTP headers. Because a list function can process a potentially huge number of rows they return data in a different way. Rather than returning a single object containing the whole response list functions must return their response in chunks. First you need to call the start function, passing in any headers that you want to return. Then you call send one or more times to return parts of your response. A typical list function will look like the code below.

function (head, req) {
    start({ "headers": { "Content-Type": "text/html" }});n
    send(header);
    while(row = getRow()) {
        data = /* process row */;
        send(row);
    }
    send(footer);
}

To process the weather data we can’t follow this simple format because we need to split each document up and display the different measurements separately. Let’s look at the code for creating the day page. The complete code is a bit too long to include in a blog post so checkout the first post in this series to find out how to get a complete copy of the code.

To start the function we load the templates and code that we need using the CouchApp macros. Next we return the appropriate Content-Type header, and then we create the object that we’ll pass to Mustache when we’ve processed everything.

function(head, req) {
    // !json templates.day
    // !json templates.head
    // !json templates.foot
    // !code vendor/couchapp/lib/mustache.js
    // !code vendor/sprintf-0.6.js
    // !code vendor/date_utils.jsn
    start({ "headers": { "Content-Type": "text/html" }});
    var stash = {
        head: templates.head,
        foot: templates.foot,
        date: req.query.startkey,
    };

Next we build a list of the documents that we’re processing so we can loop over the documents multiple times.

    var rows = [];
    while (row = getRow()) {
        rows.push(row.doc);
    }

To calculate maximum and minimum values we need to choose the first value and then run through each piece of data and see whether it is higher or lower than the current record. As the data collector of the weather station is separate to the outside sensors occasionally they lose their connection. This means that we can just pick the value in the first document as our starting value, instead we must choose the first document where the connection with the outside sensors was made.

    if(rows.length > 0) {
        for(var i=0; i<rows.length; i++) {
            if((rows[i].status &amp; 64) == 0) {
                max_temp_out = rows[i].temp_out;
                min_temp_out = rows[i].temp_out;
                max_hum_out = rows[i].hum_out;
                min_hum_out = rows[i].hum_out;
                break;
            }
        }

Now we come to the meat of the function. We loop through all of the documents and process them into a series of arrays, one for each graph that we’ll draw on the final page.

        for(var i=0; i<rows.length; i++) {
            var temp_out = null;
            var hum_out = null;
            if((rows[i].status & 64) == 0) {
                temp_out = rows[i].temp_out;
                hum_out = rows[i].hum_out;
                total_rain = total_rain + rows[i].rain;
                rainfall.push({ "time": time_text, "rain": rows[i].rain });
                wind.push({ "time": time_text, "wind_ave": rows[i].wind_ave, "wind_gust": rows[i].wind_gust });
            }n
            pressure.push({ "time": time_text, "pressure": rows[i].abs_pressure });
            temps.push({ "time": time_text, "temp_out": temp_out, "temp_in": rows[i].temp_in });
            humidity.push({ "time": time_text, "hum_in": rows[i].hum_in, ";hum_out": hum_out });
        }
    }

Lastly we take the stash, which in a bit of code I’ve not included here has the data arrays added to it, and use it to render the day template.

    send(Mustache.to_html(templates.day, stash));
    return ""
}

Let’s look at a part of the day template. The page is a fairly standard use of the Google Chart Tools library. In this first snippet we render the maximum and minimum temperature values, and a blank div that we’ll fill with the chart.

<h3>Temperature</h3>
<p>Outside: <b>Maximum:</b> <sup>o</sup>C <b>Minimum:</b> <sup>o</sup>C</p>
<p>Inside: <b>Maximum:</b> <sup>o</sup>C <b>Minimum:</b> <sup>o</sup>C</p>
<div id="tempchart_div"></div>

In the following Javascript function we build a DataTable object that we pass to the library to draw a line chart. The {{#temps}} and {{/temps}} construction is the Mustache way of looping through the temps array. We use it to dynamically write out Javascript code containing the data we want to render.

function drawTempChart() {
    var data = new google.visualization.DataTable();
    data.addColumn('string', 'Time');
    data.addColumn('number', 'Outside');
    data.addColumn('number', 'Inside');
    data.addRows([
    {{#temps}}
        ['', , ],
    {{/temps}}
        null]);
    var chart = new google.visualization.LineChart(document.getElementById('tempchart_div'));
    chart.draw(data, {width: 950, height: 240, title: 'Temperature'});
}
google.setOnLoadCallback(drawTempChart);

We now have a page that displays all the collected weather data for a single day. In the next post in this series we’ll look at how to use CouchDB’s map/reduce functions to process the data so we can display it by month and by year.

Back Garden Weather in CouchDB (Part 2)

2012-01-05T13:58:04+00:00

In my last post I described the new CouchDB-based website I have built to display the weather data collected from the weather station in my back garden. In this post I’ll describe to import the data into CouchDB and the basics of rendering a page with a CouchApp.

PyWWS writes out the raw data it collected into a series of CSV files, one per day. These are stored in two nested directory, the first being the year, the second being year-month. To collect the data I use PyWWS’s live logging mode, which consists of a process constantly running, talking to the data collector. Every five minutes it writes a new row into today’s CSV file. Another process then runs every five minutes to read the new row, and import it into the database.

Because CouchDB stores its data using an append only format you should aim to avoid unnecessary updates. The simplest way to write the import script would be to import each day’s data every five minutes. This would cause the database to balloon in size, so instead we query the database to find the last update time and import everything after than. Each update is stored as a separate document in the database, with the timestamp attribute containing the unix timestamp of the update.

The map code to get the most recent update is quite simple, we just need to emit the timestamp for each update. The reason the timestamp is emitted as the key is so we can filter the range of updates. It is also emitted as the value so we can use the timestamp in the reduce function.

function(doc) {
    emit(doc.timestamp, doc.timestamp);
}

The reduce function is a fairly simple way to calculate the maximum value of the keys. I’ve mostly included it here for completeness.

function(keys, values, rereduce) {
    if(values.length == 0) {
        return 0;
    }
    var m = values[0];
    for(var i=0; i<values.length; i++) {
        if(values[i] > m) { m = values[i]; }
    }
    return m;
}

You’ll find the import script that I use in the directory you cloned in the previous post, when you got a copy of the website.

So, we’ve got some data in our database. How do we display it on a webpage? First, let’s consider the basics of rendering a webpage.

CouchDB has two ways to display formatted data, show and list functions. Show functions allow you to format a single documents, for example a blog post. List functions allow you to format a group of documents, such as a the comments on a post. Because viewing a single piece of weather data is not interesting the weather site only uses list functions. To get started let’s create a simple Show function, as these are simpler.

CouchApp doesn’t come with a templating library, but a common one to use is Mustache. The syntax is superficially like Django templates, but in reality it is far less powerful. For a simple website like this, Mustache is perfect.

In the show directory of your CouchApp create a new file, test.js. As with the map/reduce functions this file contains an anonymous function. In this case the function takes two parameters, the document and the request obejct, and returns an object containing the response body and any headers.

function (doc, req) {
    // !json templates.records
    // !json templates.head
    // !json templates.foot
    // !code vendor/couchapp/lib/mustache.js

The function begins with some magic comments. These are commands to CouchDB which includes the referenced code or data in the function. This allows you to keep shared data separate from the functions that uses it.

The first [!json](http://guide.couchdb.org/draft/show.html#json) command causes the compiler to load the file templates/records.* and add it to a templates objects, under the records attribute.

The [!code](http://guide.couchdb.org/draft/show.html#code) command works similarly, but in loads the specified file and includes the code in your function. Here we load the Mustache library, but I have also used the function to load a javascript implementation of sprintf. You might want to load some of your own common code using this method.

    var stash = {
        head: templates.head,
        foot: templates.foot
    };
    return { body: Mustache.to_html(templates.records, stash), headers: { "Content-Type": "text/html" } };
}

Firstly we build an object containing the data we want to use in our template. As Mustache doesn’t allow you to extend templates we need to pass the header and footer HTML code in as data.

As mentioned the return type of a show function is a object containing the HTML and any HTTP headers. We only want to include the content type of the page, but you could return any HTTP header in a similar fashion. To generate the HTML we call the to_html function provided by Mustache, passing the template and the data object we prepared earlier.

Now we have data in our database and can create simple pages using a CouchApp we can move on to showing real data. In the next post I will describe the list functions use to show summarized day and month weather information.

Back Garden Weather in CouchDB (Part 1)

2011-12-02T12:00:55+00:00

When she was younger my wife wanted to be a meteorologist. That didn’t pan out, but our recent move found us with a garden, which we’ve not had before. This gave me the opportunity to buy her a weather station. I didn’t just choose any old station though, I wanted one that did wind and rain as well as the usual temperature, pressure and humidity. And, the deciding factor, a USB interface with Linux support. Fortunately the excellent PyWWS supports a range of weather stations, including the one I brought.

I’m not going to go into how I mounted the system, or configured PyWWS. That’s all covered in the documentation. PyWWS can produce a static website, but as someone who earns his living building websites I wanted something a bit better. Continuing my experiments with CouchDB I decided to build the website as a CouchApp.

As well as allowing you to query your data with Javascript, CouchDB lets you display webpages directly out of your database. If you visit welwynweather.co.uk you’ll notice that you’re redirected to a url that contains url arguments that look a lot like those used to query a view. That’s because that’s exactly what’s going on. Things become clearer when you discover that that http://www.welwynweather.co.uk is an alias for http://db.welwynweather.co.uk/_design/weather/_rewrite/. Now you can see a more complete CouchDB URL, albeit without the database name. db.welwynweather.co.uk points to an Apache reverse proxy that routes requests through to CouchDB.

Over the next few posts I’ll detail how the CouchApp works, but to get started you can clone my app and poke it yourself. Once you’ve installed the couchapp command line client simply run couchapp clone http://db.welwynweather.co.uk/_design/weather. This will give you a directory, weather, that contains a number of subdirectories including templates and views which contain the complete website.n To deploy the site to your own server you need to create a database and then run couchapp push weather http://localhost:5984/welwynweather. Visiting http://localhost:5984/welwynweather/_design/weather/_rewrite/ should show you the site. You’ll need some data though, and you can use CouchDB replication to pull my data to your server. Using Futon simply set http://db.welwynweather.co.uk/ as the replication source and your database as the destination and you’ll quickly get a complete copy of the database.

When replicating my data you currently cannot use continuous replication. When it completes replication CouchDB calls POST /_ensure_full_commit, but obviously I’ve disabled POST, PUT and DELETE on my server. This causes replication to fail and to restart from the beginning. The data will already have been copied, but CouchDB will copy it again. If you have any ideas on how to avoid this, please answer my StackOverflow question.

The website consists of four main pages. When you visit you are redirected to a page that shows the weather for the current day. Clicking on the date at the top of the page lets you also view the weather by month and by year. The daily weather pages show as much detail as is recorded by the station, in my case this is an update every five minutes. The monthly page is much the same except that the values are averaged across an hour. The yearly page is a bit different as it shows a single point for each day. An average temperature for each day is not that useful so we calculate the high and low for each day and display that.

The final page is the records page. This displays information like the highest and lowest temperature ever recorded and the heaviest rain by hour and by day. The previous three pages are all fully generated by the server. The records page is a bit different though as calculating the records in one step is a bit complicated, instead we use AJAX to load each record individually. This means we can focus on each record keeping the code simple.

In the next post I’ll discuss how I import data into CouchDB and the basics of rendering a page in a CouchApp.

If you visit the site you may find that there is no recent weather data. This is because I run PyWWS on my MythTV box. Rather than running the PC all the time the weather data only updates when a programme is being recorded, or I’m watching TV.

Programming Documentary

2011-11-25T13:00:18+00:00

I’m a huge science and engineering documentary geek. I prefer watching documentaries over all other forms of television. It doesn’t really matter what the documentary is about, I’ll usually watch it. After getting ready for my wedding I had a bit of time before I had to walk down the aisle so I watched a documentary about pilots learning to land Marine One at the White House. There probably aren’t many people who would choose to spend that time that way.

Science documentaries have experienced a renaissance over the last few years, particularly on the BBC. The long running Horizon series has been joined by a raft of other mini-series presented by Brian Cox, Alice Roberts, Marcus Du Sutoy, Jim Al-Kalili and Michael Mosely. These cover a large part of the sciences, including Chemistry, Biology and Physics. Physics in particular is regularly on our screens. Whether it’s talking about quantum mechanics or astronomy or something else it seems that Physics has never been more popular.

As someone who writes computer programmes for a living this makes me worry that your average man on the street may end up with a better understanding of quantum mechanics than they do of the computer on their desk, or in their pocket.

It wasn’t always like this. Back in 1981 the BBC ran the BBC Computer Literacy project, which attempted to teach the public to program using the BBC Micro through a ten part television series.

Clearly if a project like this was to be attempted today there would be no need for the BBC to partner with hardware manufactures. People have access to many different programmable devices, they just don’t know how to program them.

Recent programs that have focused on computers were Rory Cellan Jones’ Secret History of Social Networking and Virtual Revolution by Aleks Krotoski. Neither of these were technical documentaries, instead they focused on business, cultural and sociological impacts of computers and the internet.

It’s not that more technical aspects of computer don’t appear as part of other documentaries, recently Marcus Du Sautoy announced that he is filming a episode of Horizon on Artificial Intelligence. It won’t air until next spring, so it’s hard to comment, but I suspect it will focus on the outcome of the software rather than the process of how computers can be made to appear intelligent.

Jim Al-Kalili’s recent series on the history of electricity, Shock and Awe, ends with a section on the development of the transistor. During it, and over a picture of the LHC, he says something rather interesting.

Our computer’s have become so powerful that they are helping us to understand the universe in all its complexity.

If you don’t understand computers it’s impossible to understand how almost all modern science is done. Researches in all disiplinces need to be proficent at programming in order to anaylse their own data. Business is run on software, often which is customised to the individual requirements of the company. It boggles my mind that people can be so reliant on computers yet have so little idea of how they work.

So, what would my ideal programming documentary cover? The most obvious thing is the internet. A history of computer networking could begin at the development of the first computer networks, describe how TCP/IP divides data into packets and routes it between computers. It could move on to HTTP and HTML both of which are fundamentally simple yet apply to our everyday lives. To bring things up to date it could focus on Google and Facebook and show people the inside of a data centre. I suspect that most people have no idea where their Google search results are coming from.

I doubt that there is much demand for the updated series as long as the 10 part original, but the soon to be released Raspberry Pi machine would be an ideal way to recreate the tinkering appeal of the original BBC Micro. There’s something magical about seeing a program you’ve written appearing on the TV in your living room, rather than on the screen of your main PC. An alternative would be to provide an interpreter as part of a website so you can just type in a URL and start programming.

A documentary focussing on programming would have a difficulty that the original series never had - the fact that computing power is common place means that people are used to software created by large teams with dedicated designers. An individual with no experience can’t hope to come to close to something like that. Fortunately computers are so much more powerful today that much of the complexity that you needed to cope with can be abstracted away. Libraries like VPython make it very simple to produce complicated and impressive 3D graphics.

I’m certainly not the only person who wants to help teach the masses to program, but realistically you need an organisation like the BBC to do something that might actually make a difference. Do I think that you create a compelling and informative documentary that might inspire people to program, and give them a very basic understand of how to do it. Definitely.

Sonos Review

2011-11-18T14:08:36+00:00

Recently I purchased a basic Sonos system, and after just a couple of weeks I’m already in love with it and have more music playing in my house than ever before.

For those of you who haven’t come across Sonos before, Sonos produce a multi-room wireless music system. The system consists of a number of devices that connect to each other using a proprietary mesh network. You can buy Sonos devices that contain built in speakers, or ones that connect to your own as well as a device to link your iPhone and to join your existing network to the Sonos wireless network.

I purchased a Sonos Play:3, a Wireless Dock and ZoneBridge (all three links contain an affiliate id) so that’s what I’m reviewing here.

The Sonos Play:3 is as fairly small, unassuming, single speaker block. It contains three individual speakers while it’s larger brother, the Play:5 (affiliate link) contains five. The back has a power socket and a network port. The top has a mute button, as well as a volumn up and down rocker. The other devices are similarly spartan, yet stylish, in their design with minimal on device buttons.

First you need to plug the bridge into your network using the supplied ethernet cable. Then, after installing the PC software, or their iPhone app, you can create a Sonos network. Just follow the on screen prompts and press the ‘join’ button on the device. For each of your other Sonos devices plug them in, select “Add new device” in the software on on the app, press the ‘Join’ button (or Mute + Volumn Up on the Play:3) and the new device will be found and added the network.

The setup is supposed to be quick and straightforward, and for the first two devices it was. When I tried to add my Play:3 to the network it would repeatedly not be found. The white light on the top of the device stopped flashing, indicating that it had connected but the PC software did not find it. It’s not clear what happened, but I may have plugged it in before the previous device had finished configuring. Doing a factory reset solved the issue.

The simplest thing to play on the Sonos system is internet radio. The controller comes preloaded with a huge range of radio stations. Just select the one you want and after a short pause it’ll come out of your speaker, on the other side of the room. Not only is process of listening to the radio incredibly simple, but the sound from such a small box is amazing. I’m not an audiophile, but it was loud, clear and had plenty of bass.

To play your own music collection you need to have it available on a Windows share. I already had this set up so I just had to tell Sonos where to find it. After short while it had crawled my complete collection and I could select by artist, album, track or genre right from my iPhone. As with the radio it’s quick to start playing and the sound quality is excellent.

It was at this point that I came across the first of the few bugs I’ve found with the Sonos system. Originally I had ripped my music into Ogg Vorbis format. Then, when I got my iPhone I had to rerip it as MP3. Some of my albums have both Ogg and MP3 files of the same music, in the same directory. The Sonos player does not appear to like this, and although it can play both formats neither would appear in the controller. Where only one copy exists the files were found with no problems.

I also had some difficulties when my network was heavily loaded. While upgrading one of my pcs to the latest Ubuntu and listening to some music it skipped heavily and eventually the Play:3 crashed. Another issue is that my music is stored on my MythTV box which turns itself on and off to record tv. I forgot to lock the box so it switched itself off mid-track. Somewhat annoyingly the Play:3 stopped playing mid-track as well. I would have thought that the Sonos would have enough memory to have cached at least the whole track, if not the whole playlist.

The iPhone dock is a very useful addition to my house, if only because I just have to slip my phone in and it starts charging. It is certainly much easier to connect than a cable, and much tidier too. Unfortunately you cannot stream music from your iPhone/iPod Touch unless it is placed in the dock. This is a limitation imposed by Apple rather than Sonos, so I have to forgive them. When it’s placed in the dock any sound your device makes will be played through your speaker. This works great when you’re playing some music or a podcast through your phone, but I had a timer set on my phone which was charging while I listened some internet radio. While surprising this is just it working as expected, and you can turn off the autoplay feature.

I have my old iPhone 3G as well as much newer iPhone 4S, and if I want to keep my MythTV box off I can dock the old phone and browse its music selection and select what to listen to from the 4S. This is the real power of the Sonos concept - all your music, everywhere in your house.

The criticisms I’ve made are small points, and despite only having my system for just two weeks I already can’t imagine life without it. I’m willing to forgive the somewhat high price and am saving my pennies to buy another couple of either Play:5 or Play:3s to spread around the house.

iPhone on Holiday

2011-11-11T12:00:51+00:00

Recently I posted about upgrading my old iPhone 3G to a 64GB 4S. One of the things I was mostly looking forward to with the upgrade was the much improved camera and the ability to take video. Last week I spent some time at the Giant’s Causeway in Northern Ireland and I had plenty of opportunity to experiment with both the still and video camera.

Although it’ll never replace my DSLR the stills camera managed to take some really respectable shots. By far the biggest issue is the lack of a zoom. The colours are great, the focus is sharp but you can’t use the zoom to frame a shot. You need to move if you want to change what’s in your shot.

The video camera is also excellent. The pictures are bright, crisp and clear and it’s very easy to start recording. The small size of the iPhone makes it very easy to move the camera around. Movement is not something I’d had to think about before have previously only used a stills camera, but a static video shot gets boring very quickly.

The audio quality is quite poor, at least when you are in a windy location, and as I was on the coast it was nearly always windy. The white noise of the wind masked much of the voice that I was trying to record. Unfortunately I can’t afford a Foley artist so the wind noise remains on the final version.

To edit my travelogue there was really only one option, iMovie. As iMovie runs on the iPhone you don’t need to move your video files and get start editing straightaway. Each evening in our hotel room I could review the footage and start the editing process.

A touch interface is a natural fit for video editing work, but almost as soon as you start using iMovie you begin to realise the limitations of it. Both rearranging and deleting clips should be possible by drapping them. Instead I found it very hard to make iMovie recognise my dragging, it repeatedly selected the clip instead. Also trimming clips is difficult to do accurately because you cannot drag to frame level accuracy. While dragging the timeline around it also had the annoying habit of scrolling back to the start rather than the small adjustment I was trying to make.

The themes included with iMovie are quite nice, but as is the case with the whole app very soon after you start using them you really how limited they are. The two main problem is that when joining clips together you are limited to either a simple jump cut, cross fade or a single theme specific transition. It’s certainly important not to over use transitions, but having a few more options would be nice. You’re also extremely limited when adding text to the video, which is a key part of making a travelogue video.

I took some panorama photos using my iPhone as they’re a great way to show a scene that you just can’t capture with an ordinary camera. The first app I tried was Dermandar. This app is incredibly easy to use. You hold the camera vertically and then rotate it and the app captures pictures automatically and then stitches them together. The resulting panorama’s look great, until you try and take them off your phone. Part of the ease of use comes from the fact that it uses the video camera rather than stills camera so it’s much quicker to take the pictures. This is much lower resolution though so the resulting images are disappointing small.

The next app I tried was Autostitch panorama. Rather than the pictures being taken automatically you need to line up your photos and press a button to take a picture. When you’ve captured enough click “finish” and the app will stitch them together very quickly, producing great looking full panoramas. It’s not quite as simple to use as Dermandar, but it’s far from difficult and the results are excellent.

I’m not quite ready to leave my DSLR permanently at home, but there are certainly occasions when I will think twice about lugging it around with me.

Exceptional Catches

2011-11-02T12:00:18+00:00

Recently I was taking part in a review of some Python code. One aspect of the code really stuck out to me. It’s not a structural issue, but a minor change in programming style that can greatly improve the maintainability of the code.

The code in general was quite good, but a code snippet similar to that given below jumped right to the top of my list of things to be fixed. Why is this so bad? Let us first consider what exceptions are and why you might use them in Python.

try:
    // code
except Exception, e:
    // error handling code

Exceptions are a way of breaking out the normal program flow when an ‘exceptional’ condition arises. Typically this is used when errors occur, but exceptions can also be used as an easy way to break out of normal flow during normal but unusual conditions. In a limited set of situations it can make program flow clearer.

What does this code do though? It catches all exceptions, runs the error handling code and continues like nothing has happened. In all probability it’s only one or two errors that are expected and should be handled. Any other errors should be passed on a cause the program to actually crash so it can be debugged properly. Let’s consider the following code:

analysis_type = 1
try:
    do_analysis(analysis_typ)
except Exception, e:
    cleanup()

This code has a bug, the missing e in the do_analysis call. This will raise a NameError that will be immediately captured and hidden. Other, more complicated errors could also occur and be hidden in the same way. This sort of masking will make tracking down problems like this very difficult.

To improve this code we need to consider what errors we expect the do_analysis function to raise and what we want to handle. In the ideal case it would raise an AnalysisError and then we would catch that.

analysis_type = 1
try:
    do_analysis(analysis_typ)
except AnalysisError, e:
    cleanup()

In the improved code the NameError will pass through and be picked up immediately. It is likely that the cleanup function needs to be run whether or not an error has occurred. To do that we can move the call into a finally block.

analysis_type = 1
try:
    do_analysis(analysis_typ)
except AnalysisError, e:
    // display error message
finally:
    cleanup()

This allows us to handle a very specific error and ensure that we clean up whatever error happens. Sometimes cleaning up whatever the exception (or in the event of no exception) is required, and in this case the finally block, which is always run, is the right place for this code.

Let’s now consider a different piece of code.

try:
    do_analysis(analysis_types[index])
except KeyError:
    // display error message

We’re looking up the parameter to do_analysis in a dictionary and catching the case where index doesn’t exist. This code is also capturing too much. Not because the exception is too general, but because there is too much code in the try block.

The issue with this code is what happens if do_analysis raises a KeyError? To capture the exceptions that we’re expecting we need to only wrap the dictionary lookup in and not catch anything from the analysis call.

try:
    analysis_type = analysis_types[index]
except KeyError:
    // display error message
finally:
    do_analysis(analysis_type)

So, if I’m reviewing your code don’t be afraid to write a few extra lines in order to catch the smallest, but correct, set of exceptions.

New Delicious

2011-10-26T11:00:59+00:00

I’ve been a long time user of delicious.com, back from when it was had the hard-to-remember address of del.icio.us. Even though bookmark syncing is built into both Firefox and Chrome having your bookmarks available on a website that you can access from anywhere, and integrated into your browser using an extension is a big advantage for me.

Recently the was some cause for concern as Yahoo decided they wanted to offload the site. As it had been a Yahoo property for a long time anyone taking it on would have a big job just to extricate it from Yahoo’s infrastructure. AVOS, a new venture from the founders of YouTube took the plunge.n Several months on and the site suddenly relaunched. Superficially it looks very similar, but with a slight Web 2.0 sheen to it. The transition seems to have been handled very well given the scale of the rewrite that was required. Aside from a smattering of bugs which have been quickly squashed the biggest issue seems to have been with the people who ignored the warnings and didn’t agree to allow Yahoo to send AVOS their details.

The biggest change is the addition of stacks. These are curated sets of links on a single topic. While the old Delicious showed popular links on the frontpage the new site shows featured stacks. As you can associate an image with a stack this gives the frontpage a much more visual look. The old website was undeniably plain and the images really brighten it up.

Like many people when the Delicious troubles surfaced I immediately looked for an alternative. The most obvious choice is >Google Bookmarks</a>. Very quickly they produced a tool that imported your bookmark from Delicious making any potential switch very easy.

Should you make the switch though?

I’m glad I held off as the only advantage Google has is the closer integration with Chrome. The site is just a big list of sites with none of the social features present in Delicious. While Bookmarks might be suitable for just keeping a small list of links, Delicious’ better organization tools and style really help to manage a big list.

My biggest concern with the new site is the same as my concern with the old one. The business model, or rather, lack of. Delicious has never displayed adverts, never had any premium features and in fact has never had a way to make money (that I know of at least). Do the new owners have plan to make it profitable? I hope so, but it’s hard to imagine what feature you could add that would make people pay and taking features away is likely to create a huge backlash.

In short I’m delighted to see Delicious rise from the troubles that it faced looking stronger and better than ever.

Beating Google With CouchDB, Celery and Whoosh (Part 8)

2011-10-21T11:00:18+00:00

In the previous seven posts I’ve gone through all the stages in building a search engine. If you want to try and run it for yourself and tweak it to make it even better then you can. I’ve put the code up on GitHub. All I ask is that if you beat Google, you give me a credit somewhere.

When you’ve downloaded the code it should prove to be quite simple to get running. First you’ll need to edit settings.py. It should work out of the box, but you should change the USER_AGENT setting to something unique. You may also want to adjust some of the other settings, such as the database connection or CouchDB urls.n To set up the CouchDB views type python manage.py update_couchdb.

Next, to run the celery daemon you’ll need to type the following two commands:

python manage.py celeryd -Q retrieve
python manage.py celeryd -Q process

This sets up the daemons to monitor the two queues and process the tasks. As mentioned in a previous post two queues are needed to prevent one set of tasks from swamping the other.

Next you’ll need to run the full text indexer, which can be done with python manage.py index_update and then you’ll want to run the server using python manage.py runserver.

At this point you should have several process running not doing anything. To kick things off we need to inject one or more urls into the system. You can do this with another management command, python manage.py start_crawl http://url. You can run this command as many times as you like to seed your crawler with different pages. It has been my experience that the average page has around 100 links on it so it shouldn’t take long before your crawler is scampering off to crawl many more pages that you initially seeded it with.

So, how well does Celery work with CouchDB as a backend? The answer is that it’s a bit mixed. Certainly it makes it very easy to get started as you can just point it at the server and it just works. However, the drawback, and it’s a real show stopper, is that the Celery daemon will poll the database looking for new tasks. This polling, as you scale up the number of daemons will quickly bring your server to its knees and prevent it from doing any useful work.

The disappointing fact is that Celery could watch the _changes feed rather than polling. Hopefully this will get fixed in a future version. For now though, for anything other experimental scale installations RabbitMQ is a much better bet.

Hopefully this series has been useful to you, and please do download the code and experiment with it!

Beating Google With CouchDB, Celery and Whoosh (Part 7)

2011-10-19T11:00:16+00:00

The key ingredients of our search engine are now in place, but we face a problem. We can download webpages and store them in CouchDB. We can rank them in order of importance and query them using Whoosh but the internet is big, really big! A single server doesn’t even come close to being able to hold all the information that you would want it to - Google has an estimated 900,000 servers. So how do we scale this the software we’ve written so far effectively?

The reason I started writing this series was to investigate how well Celery’s integration with CouchDB works. This gives us an immediate win in terms of scaling as we don’t need to worry about a different backend, such as RabbitMQ. Celery itself is designed to scale so we can run celeryd daemons as many boxes as we like and the jobs will be divided amongst them. This means that our indexing and ranking processes will scale easily.

CouchDB is not designed to scale across multiple machines, but there is some mature software, CouchDB-lounge that does just that. I won’t go into how to get set this up but fundamentally you set up a proxy that sits in front of your CouchDB cluster and shards the data across the nodes. It deals with the job of merging view results and managing where the data is actually stored so you don’t have to. O’Reilly’s CouchDB: The Definitive Guide has a chapter on clustering that is well worth a read.

Unfortunately while Woosh is easy to work with it’s not designed to be used on a large scale. Indeed if someone was crazy enough to try to run the software we’ve developed in this series they might be advised to replace Whoosh with Solr. Solr is a lucene-based search server which provides an HTTP interface to the full-text index. Solr comes with a sharding system to enable you to query an index that is too large for a single machine.

So, with our two data storage tools providing HTTP interface and both having replication and sharding either built in or as available as a proxy the chances of being able to scale effectively are good. Celery should allow the background tasks that are needed to run a search engine can be scaled, but the challenges of building and running a large scale infrastructure are many and I would not claim that these tools mean success is guarenteed!n In the final post of this series I will discuss what I’ve learnt about running Celery with CouchDB, and with CouchDB in general. I’ll also describe how to download and run the complete code so you can try these techniques for yourself.

Read part 8.

iPhone 4S

2011-10-17T11:00:53+00:00

This weekend I joined the hysterical masses and upgraded my increasingly ancient iPhone 3G to a shiny new 64GB iPhone 4S. Except that it was actually a bit of an anticlimax. I went into my local O2 shop at about 10:30am on Saturday morning, the day after the launch, and purchased a phone. No queueing, no raging hoards. I didn’t even have to shove a granny out of the way to get one. However, after handing over my credit card while cringing at the expense it was back home to enjoy the famous Apple unboxing experience.

I wish I’d never upgraded my 3G to iOS 4.2. Up until that point it was a great phone. Afterwards it was slow and applications would repeated crash on start up. Did I mention it was slow?

It’s hard to express just how much quicker the 4S is compared to my 3G. Often just typing my the passcode would be too quick for the 3G and it would miss one of the numbers forcing me to go back. No danger of this with the 4G though. Application starting, browsing the web, taking photos are all super speedy.

Although it’s the same as the iPhone 4 the screen is still incredible. It’s so bright and sharp it’s really a joy to use. It really comes into its own when browsing webpages that are designed for bigger screens. The extra detail really helps you to work out where to zoom in.

The camera is also much improved, and I’m sure the addition of video compared to my old 3G will come in useful. I brought iMovie and that seems like an easy way to put together some short videos of my holidays. I haven’t had much of a chance to experiment with this aspect of the phone properly yet, but I can imagine that where carrying equipment is a problem it will replace my DSLR as my primary camera. I’m sure I’ll have more to say on this topic when I’ve been out and about for a while.

I don’t intend to make this an in depth review of the phone, there are many hundreds of better places that you could go to for that. One feature that I can’t not talk about is Siri. I struggle to see why, for a non-blind user, you would use it after playing with it for a bit. If you are blind then dictating is clearly a huge help, and the biggest surprise for me is the quality of the voice recognition. I don’t have a strong accent so I suppose that if it didn’t work for me then it wouldn’t work for anyone. Still though, for someone who hasn’t used voice recognition for ten years it’s amazing just how far it has come.

There’s lots of talk online about the funny answers that Siri comes up with if you ask it question like “What’s the meaning of life?” It’s certainly to Apple’s credit that they’ve given it so much personality and it’s not just a bland robot. These questions are not a long term use case though.

Where Siri really succeeds is as an interface to Wolfram Alpha. Like most people when Wolfram Alpha was launched I played with it for a bit and then forgot about it. The ability to say things like “What’s the distance from the Earth to the Moon?” and have Siri return the correct answer is quite amazing. People are used to typing a few words into Google but Wolfram Alpha needs a bit more than that. A voice interface seems so much more natural and is probably Siri’s killer application.

Unfortunately as I’m in the UK Siri doesn’t do mapping or business search for me. Roll on 2012 when that is supposed to be added.

It’s early days yet, but I’ve fallen in love with my phone all over again.

Beating Google With CouchDB, Celery and Whoosh (Part 6)

2011-10-13T11:00:30+00:00

We’re nearing the end of our plot to create a Google-beating search engine (in my dreams at least) and in this post we’ll build the interface to query the index we’ve built up. Like Google the interface is very simple, just a text box on one page and a list of results on another.

To begin with we just need a page with a query box. To make the page slightly more interesting we’ll also include the number of pages in the index, and a list of the top documents as ordered by our ranking algorithm.

In the templates on this page we reference base.html which provides the boiler plate code needed to make an HTML page.

{% extends "base.html" %}
{% block body " %}
    <form action="/search" method="get">
        <input name="q" type="text">
        <input type="submit">
    </form>
    <hr>
    <p>{{ doc_count }} pages in index.</p>
    <hr>
    <h2>Top Pages</h2>
    <ol>
    {% for page in top_docs %}
        <li><a href="{{ page.url }}">{{ page.url }}</a> - {{ page.rank }}</li>
    {% endfor %}
    </ol>
{% endblock" %}

To show the number of pages in the index we need to count them. We’ve already created an view to list Pages by their url and CouchDB can return the number of documents in a view without actually returning any of them, so we can just get the count from that. We’ll add the following function to the Page model class.

    @staticmethod
    def count():
        r = settings.db.view("page/by_url", limit=0)
        return r.total_rows

We also need to be able to get a list of the top pages, by rank. We just need to create view that has the rank as the key and CouchDB will sort it for us automatically.

With all the background pieces in place the Django view function to render the index is really very straightforward.

def index(req):
    return render_to_response("index.html", { "doc_count": Page.count(), "top_docs": Page.get_top_by_rank(limit=20) })

Now we get to the meat of the experiment, the search results page. First we need to query the index.

def search(req):
    q = QueryParser("content", schema=schema).parse(req.GET["q"])

This parses the user submitted query and prepares the query ready to be used by Whoosh. Next we need to pass the parsed query to the index.

    results = get_searcher().search(q, limit=100)

Hurrah! Now we have list of results that match our search query. All that remains is to decide what order to display them in. To do this we normalize the score returned by Whoosh and the rank that we calculated, and add them together.

    if len(results) > 0:
        max_score = max([r.score for r in results])
        max_rank = max([r.fields()["rank"] for r in results])

To calculate our combined rank we normalize the score and the rank by setting the largest value of each to one and scaling the rest appropriately.

        combined = []
        for r in results:
            fields = r.fields()
            r.score = r.score/max_score
            r.rank = fields["rank"]/max_rank
            r.combined = r.score + r.rank
            combined.append(r)

The final stage is to sort our list by the combined score and render the results page.

        combined.sort(key=lambda x: x.combined, reverse=True)
    else:
        combined = []
    return render_to_response("results.html", { "q": req.GET["q"], "results": combined })

The template for the results page is below.

{% extends "base.html" %}
{% block body %}
    <form action="/search" method="get">
        <input name="q" type="text" value="{{ q }}">
        <input type="submit">
    </form>
    {% for result in results|slice:":20"%}
        <p>
            <b><a href="{{ result.url }}">{{ result.title|safe }}</a></b> ({{ result.score }}, {{ result.rank }}, {{ result.combined }})<br>
            {{ result.desc|safe }}
        </p>
    {% endfor %}
{% endblock %}

So, there we have it. A complete web crawler, indexer and query website. In the next post I’ll discuss how to scale the search engine.

Read part 7.

Beating Google With CouchDB, Celery and Whoosh (Part 5)

2011-10-11T11:00:16+00:00

In this post we’ll continue building the backend for our search engine by implementing the algorithm we designed in the last post for ranking pages. We’ll also build a index of our pages with Whoosh, a pure-Python full-text indexer and query engine.

To calculate the rank of a page we need to know what other pages link to a given url, and how many links that page has. The code below is a CouchDB map called page/links_to_url. For each page this will output a row for each link on the page with the url linked to as the key and the page’s rank and number of links as the value.

function (doc) {
    if(doc.type == "page") {
        for(i = 0; i < doc.links.length; i++) {
            emit(doc.links[i], [doc.rank, doc.links.length]);
        }
    }
}

As before we’re using a Celery task to allow us to distribute our calculations. When we wrote the find_links task we called calculate_rank with the document id for our page as the parameter.

@task
def calculate_rank(doc_id):
    page = Page.load(settings.db, doc_id)

Next we get a list of ranks for the page’s that link to this page. This static method is a thin wrapper around the page/links_to_url map function given above.

    links = Page.get_links_to_url(page.url)

Now we have the list of ranks we can calculate the rank of this page by dividing the rank of the linking page by the number of links and summing this across all the linking pages.

    rank = 0
    for link in links:
        rank += link[0] / link[1]

To prevent cycles (where A links to B and B links to A) from causing an infinite loop in our calculation we apply a damping factor. This causes the value of the link to decline by 0.85 and combined with the limit later in the function will force any loops to settle on a value.

    old_rank = page.rank
    page.rank = rank * 0.85

If we didn’t find any links to this page then we give it a default rank of 1/number_of_pages.

    if page.rank == 0:
        page.rank = 1.0/settings.db.view("page/by_url", limit=0).total_rows

Finally we compare the new rank to the previous rank in our system. If it has changed by more than 0.0001 then we save the new rank and cause all the pages linked to from our page to recalculate their rank.

    if abs(old_rank - page.rank) > 0.0001:
        page.store(settings.db)
        for link in page.links:
            p = Page.get_id_by_url(link, update=False)
            if p is not None:
                calculate_rank.delay(p)

This is a very simplistic implementation of a page rank algorithm. It does generate a useful ranking of pages, but the number of queued calculate_rank tasks explodes. In a later post I’ll discuss how this could be made rewritten to be more efficient.

Whoosh is a pure-Python full text search engine. In the next post we’ll look at querying it, but first we need to index the pages we’ve crawled.

The first step with Whoosh is to specify your schema. To speed up the display of results we store the information we need to render the results page directly in the schema. For this we need the page title, url and description. We also store the score given to the page by our pagerank-like algorithm. Finally we add the page text to the index so we can query it. If you want more details, the Whoosh documentation is pretty good.

from whoosh.fields import *n
schema = Schema(title=TEXT(stored=True), url=ID(stored=True, unique=True), desc=ID(stored=True), rank=NUMERIC(stored=True, type=float), content=TEXT)

CouchDB provides an interface for being informed whenever a document in the database changes. This is perfect for building an index.

Our full-text indexing daemon is implemented as a Django management command so there is some boilerplate code required to make this work.

class Command(BaseCommand):
    def handle(self, **options):
        since = get_last_change()
        writer = get_writer()

CouchDB allows you to get all the changes that have occurred since a specific point in time (using a revision number). We store this number inside the Whoosh index directory, and accessing it using the get_last_change and set_last_change functions. Our access to the Whoosh index is through a IndexWriter object, again accessed through an abstraction function.

Now we enter an infinite loop and call the changes function on our CouchDB database object to get the changes.

        try:
            while True:
                changes = settings.db.changes(since=since)
                since = changes["last_seq"]
                for changeset in changes["results"]:
                    try:
                        doc = settings.db[changeset["id"]]
                    except couchdb.http.ResourceNotFound:
                        continue

In our database we store robots.txt files as well as pages, so we need to ignore them. We also need to parse the document so we can pull out the text from the page. We do this with the BeautifulSoup library.

                    if "type" in doc and doc["type"] == "page":
                        soup = BeautifulSoup(doc["content"])
                        if soup.body is None:
                            continue

On the results page we try to use the meta description if we can find it.

                        desc = soup.findAll('meta', attrs={ "name": desc_re })

Once we’ve got the parsed document we update our Whoosh index. The code is a little complicated because we need to handle the case where the page doesn’t have a title or description, and that we search for the title as well as the body text of the page. The key element here is text=True which pulls out just the text from a node and strips out all of the tags.

                        writer.update_document(
                                title=unicode(soup.title(text=True)[0]) if soup.title is not None and len(soup.title(text=True)) > 0 else doc["url"],
                                url=unicode(doc["url"]),
                                desc=unicode(desc[0]["content"]) if len(desc) > 0 and desc[0]["content"] is not None else u"",
                                rank=doc["rank"],
                                content=unicode(soup.title(text=True)[0] + "\n" + doc["url"] + "\n" + "".join(soup.body(text=True)))
                            )

Finally we update the index and save the last change number so next time the script is run we continue from where we left off.

                    writer.commit()
                    writer = get_writer()
                set_last_change(since)
        finally:
            set_last_change(since)

In the next post I’ll discuss how to query the index, sort the documents by our two rankings and build a simple web interface.

Read part 6.

Beating Google With CouchDB, Celery and Whoosh (Part 4)

2011-10-06T11:00:26+00:00

In this series I’m showing you how to build a webcrawler and search engine using standard Python based tools like Django, Celery and Whoosh with a CouchDB backend. In previous posts we created a data structure, parsed and stored robots.txt and stored a single webpage in our document. In this post I’ll show you how to parse out the links from our stored HTML document so we can complete the crawler, and we’ll start calculating the rank for the pages in our database.

There are several different ways of parsing out the links in a given HTML document. You can just use a regular expression to pull the urls out, or you can use a more complete but also more complicated (and slower) method of parsing the HTML using the standard Python htmlparser library, or the wonderful Beautiful Soup. The point of this series isn’t to build a complete webcrawler, but to show you the basic building blocks. So, for simplicity’s sake I’ll use a regular expression.

link_single_re = re.compile(r"<a[^>]+href='([^']+)'")
link_double_re = re.compile(r'<a[^>]+href="([^"]+)"')

All we need to look for an href attribute in an a tag. We’ll use two regular expressions to handle single and double quotes, and then build a list containing all the links in the document.

@task
def find_links(doc_id):
    doc = Page.load(settings.db, doc_id)
    raw_links = []
    for match in link_single_re.finditer(doc.content):
        raw_links.append(match.group(1))
        for match in link_double_re.finditer(doc.content):
            raw_links.append(match.group(1))

Once we’ve got a list of the raw links we need to process them into absolute urls that we can send back to the retrieve_page task we wrote earlier. I’m cutting some corners with processing these urls, in particular I’m not dealing with base tags.

    doc.links = []
    for link in raw_links:
        if link.startswith("#"):
            continue
        elif link.startswith("http://") or link.startswith("https://"):
            pass
        elif link.startswith("/"):
            parse = urlparse(doc["url"])
            link = parse.scheme + "://" + parse.netloc + link
        else:
            link = "/".join(doc["url"].split("/")[:-1]) + "/" + link
        doc.links.append(unescape(link.split("#")[0]))
    doc.store(settings.db)

Once we’ve got our list of links and saved the modified document we then need to trigger the next series of steps to occur. We need to calculate the rank of this page, so we trigger that task and then we step through each page that we linked to. If we’ve already got a copy of the page then we want to recalculate its rank to take into account the rank of this page (more on this later) and if we don’t have a copy then we queue it up to be retrieved.

    calculate_rank.delay(doc.id)
    for link in doc.links:
        p = Page.get_id_by_url(link, update=False)
        if p is not None:
            calculate_rank.delay(p)
        else:
            retrieve_page.delay(link)

We’ve now got a complete webcrawler. We can store webpages and robots.txt files. Given a starting URL our crawler will set about parsing pages to find out what they link to and retrieve those pages as well. Given enough time you’ll end up with most of the internet on your harddisk!

When we come to write the website to query the information we’ve collected we’ll use two numbers to rank pages. First we’ll use the a value that ranks pages base on the query used, but we’ll also use a value that ranks pages based on their importance. This is the same method used by Google, known as Page Rank.

Pank Rank is a measure of how likely you are to end up on a given page by clicking on a random link anywhere on the internet. The Wikipedia article goes into some detail on a number of ways to calculate it, but we’ll use a very simple iterative algorithm.

When created, a page is given a rank equal to 1/number of pages. Each link that is found on a newly crawled page then causes the rank of the destination page to be calculated. In this case the rank of a page is the sum of the ranks of the pages that link to it, divided by the number of links on those pages, multiplied by a dampening factor (I use 0.85, but this could be adjusted.) If a page has a rank of 0.25 and has five links then each page linked to gains 0.05*0.85 rank for that link. If the change in rank of the page when recalculated is significant then the rank of all the pages it links to are recalculated.

In this post we’ve completed the web crawler part of our search engine and discussed how to rank pages in importance. In the next post we’ll implement this ranking and also create a full text index of the pages we have crawled.

Read part 5.

Beating Google With CouchDB, Celery and Whoosh (Part 3)

2011-10-04T11:00:18+00:00

In this series I’ll show you how to build a search engine using standard Python tools like Django, Whoosh and CouchDB. In this post we’ll start crawling the web and filling our database with the contents of pages.

One of the rules we set down was to not request a page too often. If, by accident, we try to retrieve a page more than once a week then don’t want that request to actually make it to the internet. To help prevent this we’ll extend the Page class we created in the last post with a function called get_by_url. This static method will take a url and return the Page object that represents it, retrieving the page if we don’t already have a copy. You could create this as an independent function, but I prefer to use static methods to keep things tidy.

We only actually want to retrieve the page from the internet in one of the three tasks the we’re going to create so we’ll give get_by_url a parameter, update that enables us to return None if we don’t have a copy of the page.

@staticmethod
def get_by_url(url, update=True):
    r = settings.db.view("page/by_url", key=url)
    if len(r.rows) == 1:
        doc = Page.load(settings.db, r.rows[0].value)
        if doc.is_valid():
            return doc
    elif not update:
        return None
    else:
        doc = Page(url=url)
        doc.update()
        return doc

The key line in the static method is doc.update(). This calls the function to retrieves the page and makes sure we respect the robots.txt file. Let’s look at what happens in that function.

def update(self):
    parse = urlparse(self.url)

We need to split up the given URL so we know whether it’s a secure connection or not, and we need to limit our connects to each domain so we need get that as well. Python has a module, urlparse, that does the hard work for us.

    robotstxt = RobotsTxt.get_by_domain(parse.scheme, parse.netloc)
    if not robotstxt.is_allowed(parse.netloc):
        return False

In the previous post we discussed parsing the robots.txt file and here we make sure that if we’re not allowed to index a page, then we don’t

    while cache.get(parse.netloc)
        is not None:
            time.sleep(1)
            cache.set(parse.netloc, True, 10)

As with the code to parse robots.txt files we need to make sure we don’t access the same domain too often.

    req = Request(self.url, None, { "User-Agent": settings.USER_AGENT })
    resp = urlopen(req)
    if not resp.info()["Content-Type"].startswith("text/html"):
        return
    self.content = resp.read().decode("utf8")
    self.last_checked = datetime.now()
    self.store(settings.db)

Finally, once we’ve checked we’re allowed to access a page and haven’t accessed another page on the same domain recently we use the standard Python tools to download the content of the page and store it in our database.n Now we can retrieve a page we need to add it to the task processing system. To do this we’ll create a Celery task to retrieve the page. The task just needs to call the get_by_url static method we created earlier and then, if the page is downloaded trigger a second task to parse out all of the links.

@task
def retrieve_page(url):
    page = Page.get_by_url(url)
    if page is None:
        return
    find_links.delay(page.id)

You might be asking why the links aren’t parsed immediately after retrieving the page. They certainly could be, but a key goal was to enable the crawling process to scale as much as possible. Each page crawled has, based on the pages I’ve crawled so far, around 100 links on it. As part of the find_links task a new retrieve_task is created. This quickly swamps the tasks to perform other tasks like calculating the rank of a page and prevents them from being processed.

Celery provides the tools to ensure that a subset of message are processed in a timely manner, called Queues. Tasks can be assigned to different queues and daemons can be made to watch a specific set of queues. If you have a Celery daemon that only watches the queue used by your high priority tasks then those tasks will always be processed quickly.

We’ll use two queues, one for retrieving the pages and another for processing them. First we need to tell Celery about the queues (we also need to include the default celery queue here) and then we create a router class. The router looks at the task name and decides which queue to put it into. Your routing code could be very complicated, but ours is very straightforward.

CELERY_QUEUES = {
    "retrieve": {
        "exchange": "default",
        "exchange_type": "direct",
        "routing_key": "retrieve"
    },
    "process": {
        "exchange": "default",
        "exchange_type": "direct",
        "routing_key": "process "
    },
    "celery": {
        "exchange": "default",
        "exchange_type": "direct",
        "routing_key": "celery"
    }
}

class MyRouter(object):
    def route_for_task(self, task, args=None, kwargs=None):
        if task == "crawler.tasks.retrieve_page":
            return { "queue": "retrieve" }
        else:
            return { "queue": "process" }

CELERY_ROUTES = (MyRouter(), )

The final step is to allow the crawler to be kicked off by seeding it with some URLs. I’ve previously posted about how to create a Django management command and they’re a perfect fit here. The command takes one argument, the url, and creates a Celery task to retrieve it.

class Command(BaseCommand):
    def handle(self, url, **options):
        retrieve_page.delay(url)

We’ve now got a web crawler that is almost complete. In the next post I’ll discuss parsing links out of the HTML, and we’ll look at calculating the rank of each page.

Read part 4.

Beating Google With CouchDB, Celery and Whoosh (Part 2)

2011-09-29T11:00:18+00:00

In this series I’ll show you how to build a search engine using standard Python tools like Django, Whoosh and CouchDB. In this post we’ll begin by creating the data structure for storing the pages in the database, and write the first parts of the webcrawler.

CouchDB’s Python library has a simple ORM system that makes it easy to convert between the JSON objects stored in the database and a Python object.

To create the class you just need to specify the names of the fields, and their type. So, what do a search engine need to store? The url is an obvious one, as is the content of the page. We also need to know when we last accessed the page. To make things easier we’ll also have a list of the urls that the page links to. One of the great advantages of a database like CouchDB is that we don’t need to create a separate table to hold the links, we can just include them directly in the main document. To help return the best pages we’ll use a page rank like algorithm to rank the page, so we also need to store that rank. Finally, as is good practice on CouchDB we’ll give the document a type field so we can write views that only target this document type.

class Page(Document):
    type = TextField(default="page")
    url = TextField()
    content = TextField()
    links = ListField(TextField())
    rank = FloatField(default=0)
    last_checked = DateTimeField(default=datetime.now)

That’s a lot of description for not a lot of code! Just add that class to your models.py file. It’s not a normal Django model, but we’re not using Django models in this project so it’s the right place to put it.

We also need to keep track of the urls that we are and aren’t allowed to access. Fortunately for us Python comes with a class, RobotFileParser which handles the parsing of the file for us. So, this becomes a much simpler model. We just need the domain name, and a pickled RobotFileParser instance. We also need to know whether we’re accessing an http or https and we’ll give it type field to distinguish it from the Page model.

class RobotsTxt(Document):
    type = TextField(default="robotstxt")
    domain = TextField()
    protocol = TextField()
    robot_parser_pickle = TextField()

We want to make the pickle/unpickle process transparent so we’ll create a property that hides the underlying pickle representation. CouchDB can’t store the binary pickle value, so we also base64 encode it and store that instead. If the object hasn’t been set yet then we create a new one on the first access.

    def _get_robot_parser(self):
        if self.robot_parser_pickle is not None:
            return pickle.loads(base64.b64decode(self.robot_parser_pickle))
        else:
            parser = RobotFileParser()
            parser.set_url(self.protocol + "://" + self.domain + "/robots.txt") self.robot_parser = parser
            return parser

    def _set_robot_parser(self, parser):
        self.robot_parser_pickle = base64.b64encode(pickle.dumps(parser))
    robot_parser = property(_get_robot_parser, _set_robot_parser)

For both pages and robots.txt files we need to know whether we should reaccess the page. We’ll do this by testing whether the we accessed the file in the last seven days of not. For Page models we do this by adding the following function which implements this check.

    def is_valid(self):
        return (datetime.now() - self.last_checked).days < 7

For the RobotsTxt we can take advantage of the last modified value stored in the RobotFileParser that we’re wrapping. This is a unix timestamp so the is_valid function needs to be a little bit different, but follows the same pattern.

    def is_valid(self):
        return (time.time() - self.robot_parser.mtime()) < 7*24*60*60

To update the stored copy of a robots.txt we need to get the currently stored version, read a new one, set the last modified timestamp and then write it back to the database. To avoid hitting the same server too often we can use Django’s cache to store a value for ten seconds, and sleep if that value already exists.

    def update(self):
        while cache.get(self.domain) is not None: time.sleep(1)
        cache.set(self.domain, True, 10)
        parser = self.robot_parser
        parser.read()
        parser.modified()
        self.robot_parser = parser
        self.store(settings.db)

Once we’ve updated the stored file we need to be able to query it. This function just passes the URL being tested through to the underlying model along with our user agent string.

def is_allowed(self, url):
    return self.robot_parser.can_fetch(settings.USER_AGENT, url)

The final piece in our robots.txt puzzle is a function to pull the write object out of the database. We’ll need a view that has the protocol and domain for each file as the key.

@staticmethod
def get_by_domain(protocol, domain):
    r = settings.db.view("robotstxt/by_domain", key=[protocol, domain])

We query that mapping and if it returns a value then we load the object. If it’s still valid then we can return right away, otherwise we need to update it.

    if len(r) > 0:
        doc = RobotsTxt.load(settings.db, r.rows[0].value)
        if doc.is_valid():
        return doc

If we’ve never loaded this domain’s robots.txt file before then we need to create a blank object. The final step is to read the file and store it in the database.

    else:
        doc = RobotsTxt(protocol=protocol, domain=domain)n doc.update()
        return doc

For completeness, here is the map file required for this function.

    function (doc) {
        if(doc.type == "robotstxt") {
            emit([doc.protocol, doc.domain], doc._id);
        }
    }

In this post we’ve discussed how to represent a webpage in our database as well as keep track of what paths we are and aren’t allowed to access. We’ve also seen how to retrieve the robots.txt files and update them if they’re too old.

Now that we can test whether we’re allowed to access a URL in the next post in this series I’ll show you how to begin crawling the web and populating our database.

Read part 3.

Beating Google With CouchDB, Celery and Whoosh (Part 1)

2011-09-27T11:00:44+00:00

Ok, let’s get this out of the way right at the start - the title is a huge overstatement. This series of posts will show you how to create a search engine using standard Python tools like Django, Celery and Whoosh with CouchDB as the backend.

Celery is a message passing library that makes it really easy to run background tasks and to spread them across a number of nodes. The most recent release added the NoSQL database CouchDB as a possible backend. I’m a huge fan of CouchDB, and the idea of running both my database and message passing backend on the same software really appealed to me. Unfortunately the documentation doesn’t make it clear what you need to do to get CouchDB working, and what the downsides are. I decided to write this series partly to explain how Celery and CouchDB work, but also to experiment with using them together.

In this series I’m going to talk about setting up Celery to work with Django, using CouchDB as a backend. I’m also going to show you how to use Celery to create a web-crawler. We’ll then index the crawled pages using Whoosh and use a PageRank-like algorithm to help rank the results. Finally, we’ll attach a simple Django frontend to the search engine for querying it.

Let’s consider what we need to implement for our webcrawler to work, and be a good citizen of the internet. First and foremost is that we must be read and respect robots.txt. This is a file that specifies what areas of a site crawlers are banned from. We must also not hit a site too hard, or too often. It is very easy to write a crawler than repeatedly hits a site, and requests the same document over and over again. These are very big no-noes. Lastly we must make sure that we use a custom User Agent so our bot is identifiable.

We’ll divide the algorithm for our webcrawler into three parts. Firstly we’ll need a set of urls. The crawler picks a url, retrieves the page then store it in the database. The second stage takes the page content, parses it for links, and adds the links to the set of urls to be crawled. The final stage is to index the retrieved text. This is done by watching for pages that are retrieved by the first stage, and adding them to the full text index.

Celery’s allows you to create ‘tasks’. These are units of work that are triggered by a piece of code and then executed, after a period of time, on any node in your system. For the crawler we’ll need two seperate tasks. The first retrieves and stores a given url. When it completes it will triggers a second task, one that parses the links from the page. To begin the process we’ll need to use an external command to feed some initial urls into the system, but after that it will continuously crawl until it runs out of links. A real search engine would want to monitor its index for stale pages and reload those, but I won’t implement that in this example.

I’m going to assume that you have a decent level of knowledge about Python and Django, so you might want to read some tutorials on those first. If you’re following along at home, create yourself a blank Django project with a single app inside. You’ll also need to install django-celery, the CouchDB Python library, and have a working install of CouchDB available.

Read part 2.

Crowd Sourcing Mapping

2011-05-20T12:59:15+00:00

Recently Google announced that they were making their crowd sourcing mapping tools available to users in the United States. This tool lets uses edit Google Maps, adding businesses and even roads, railways and rivers. This raises interesting questions about whether wisdom of the crowd can be applied to data that requires a high degree of accuracy.

Open Street Map has been doing this since 2004, and has put together an amazing resource of free map data, but only recently has Google begun to allow people to edit its maps for large parts of the world.

Accurate mapping data is terribly important. While the majority of Google Maps queries are likely to be “how do I get from my house to my aunt’s?” some are much more important. A war was almost caused when the border between Nicaraguan and Costa Rica was incorrectly placed. While a war is a little far-fetched, it’s not hard to imagine how a mistake on map could cost someone’s life in a medical emergency.

Originally Open Street Map required budding cartographers to get out with their GPS devices and manually record their position. With the explosion of satellite* mapping information creating maps just involves sitting at your webbrowser and clicking. But how accurate are the satellite images? It’s easy to find a road on Google Maps that disagrees with the imagines underneath it. It’s not so easy to work out which one is correct. The height and angle of the terrain, the location of the plane and other processing errors may make the satellite imagery not line up with GPS data.

Like most crowd sourced data sources Google’s Map Maker and Open Streetmap have a voting process so vandalism will be weeded out, but for people sitting at their computers looking at the same, possibly misleading photos, it won’t improve the accuracy of the changes.

Without knowing more about the collection process for the satellite imagery it’s hard to know what level of accuracy they have. Presumably it’s pretty good, but we need to decide how accurate maps need to be. When you’re looking a country level map even a kilometer here or there doesn’t matter too much, but get down to the level of a walker and suddenly centimeters become important.

What’s more important, masses of mapping information or accurate maps? I’m not sure, but I think it’s probably the latter.

If you want to see the mass of data that is being created by the crowd, Map Maker Pulse is a fascinating and hypnotic site to visit.

The final issue is that of licensing. Open Street Map is very clear that any time you spend improving their maps is rewarded by your work being released under an open source license. Google’s terms of service require you to given them a license to do whatever they like to your work. They do not need to make your changes available for others to use. Google is asking you to do high precision work for free and then taking that work and locking it inside in Google Maps, and that doesn’t seem like a fair trade to me.

Hopefully no one is actually using imagery from a satellite for mapping, but rather photographs taken from a
plane.

The Importance Of Documentation

2011-04-08T13:04:09+00:00

Recently I’ve been working a couple of open source projects and as part of them I’ve been using some libraries. In order to use a library though, you need to understand how it is designed, what function calls are available and what those functions do. The two libraries I’ve been using are Qt and libavformat, which is part of FFmpeg and they show two ends of the documentation spectrum.

Now, it’s important to note that Qt is a massive framework owned by Nokia, with hundreds of people working on it full-time including a number of people dedicated to documentation. FFmpeg on the other hand is a purely volunteer effort with only a small number of developers working on it. Given the complicated nature of video encoding to have a very stable and feature-full library such as FFmpeg available as open source software is almost a miracle. Comparing the levels of documentation between these two projects is very unfair, but it serve as a useful example of where documentation can sometimes be lacking across all types of projects, both open and closed source.n So, lets look at what documentation it is important to write by considering how you might approach using a new library.

When you start using some code that you’ve not interacted with before the first thing that you need is to get a grasp on the concepts underlying the library. Some libraries are functional, some object orientated. Some use callbacks, others signals and slots. You also need to know the top level groupings of the elements in the library so you can narrow your focus that parts of the library you actually want to use.

Qt’s document starts in a typical fashion, with a tutorial. This gives you a very short piece of code that gets you up and running quickly. It then proceeds to describe, line by line, how the code works and so introduces you to the fundamental concepts used in the library. FFmpeg takes a similar approach, and links to a tutorial. However, the tutorial begins with a big message about it being out of date. How much do you trust the out of date tutorial?

Once you’ve a grasp of the fundamental design decisions that were taken while building the library, you’ll need to find out what functions you need to call or objects you need to create to accomplished your goal. Right at the top of the menu the QT documentation has links to class index, function index and modules. These let you easily browse the contents of the library and delve into deeper documentation. Doxygen is often used to generate documentation for an API, and it seem to be the way FFmpeg is documented. Their frontpage contains… nothing. It’s empty.

Finally, you’ll need to know what arguments to pass to a function and what to expect in return. This is probably the most common form of documentation to write so you probably (hopefully?) already write it. Despite my earlier criticisms, FFmpeg does get this right and most of the function describe what you’re supposed to pass into the function. With this sort of documentation it’s important to strike a balance. You need to write enough documentation such that people can call your function and have it work first time, but you don’t want to write too much so that it takes ages to get to grips with or replicates what you could find out by reading the code.n Some people hold on to the belief that the code is the ultimate documentation. Certainly writing self-documenting code is a worthy goal, but there are other levels of documentation that are needed before someone could read and the code and understand it well enough for it to be self-documenting. So, next time you’re writing a library make sure you consider:

How do people get started? How do people navigate through the code? and, how do people work out how to call
my functions?

Deadlock On Exit With PySide And QFileSystemWatcher

2011-03-14T13:04:09+00:00

Last year Nokia started developing their own Python bindings for Qt, PySide, when they couldn’t persuade Riverbank Computing to relicense PyQt under a more liberal license. While developing DjangoDE I made the choice of which library to use configurable. When running under PyQt everything worked fine, but when using PySide the program hung on exit.

Using gdb to see where it was hanging points to QFileSystemWatcher, which has the following comment in the destructor.

Note: To avoid deadlocks on shutdown, all instances of QFileSystemWatcher need to be destroyed before QCoreApplication. Note that passing QCoreApplication::instance() as the parent object when creating QFileSystemWatcher is not sufficient.

The following code will demonstrate the issue.

import sys
#from PyQt4 import QtGui
from PySide import QtGui
app = QtGui.QApplication(sys.argv)
file_browser = QtGui.QTreeView()
file_model = QtGui.QFileSystemModel()
file_model.setRootPath("/")
file_browser.setModel(file_model)
file_browser.show()
sys.exit(app.exec_())

As the comment says, we need to ensure that the QFileSystemWatcher object, which is created by QFileSystemModel, is destroyed before QApplication. To do this we can connect to the lastWindowClosed and ensure that the QFileSystemModel is fully destroyed.

import gcn
def app_quit():
     global file_browser
     file_browser = None
     gc.collect()
app.lastWindowClosed.connect(app_quit)

It’s not clear why this code would work on PyQt and not PySide, but it is clearly related to the order the objects are deleted. Given the comment in the Qt documentation though you should probably not rely on it working on PyQt and ensure yourself that the QApplication is the last Qt object to be destroyed.

Cleaning Your Django Project With PyLint And Buildbot

2011-03-07T13:04:09+00:00

There are a number of tools for checking whether your Python code meets a coding standard. These include pep8.py, PyChecker and PyLint. Of these, PyLint is the most comprehensive and is the tool which I prefer to use as part of my buildbot checks that run on every commit.

PyLint works by parsing the Python source code itself and checking things like using variables that aren’t defined, missing doc strings and a large array of other checks. A downside of PyLint’s comprehensiveness is that it runs the risk of generating false positives. As it parses the source code itself it struggles with some of Python’s more dynamic features, in particular metaclasses, which, unfortunately, are a key part of Django. In this post I’ll go through the changes I make to the standard PyLint settings to make it more compatible with Django.

disable=W0403,W0232,E1101

This line disables a few problems that are picked up entirely. W0403 stops relative imports from generating a warning, whether you want to disable these or not is really a matter of personal preference. Although I appreciate why there is a check for this, I think this is a bit too picky. W0232 stops a warning appearing when a class has no __init__ method. Django models will produce this warning, but because they’re metaclasses there is nothing wrong with them. Finally, E1101 is generated if you access a member variable that doesn’t exist. Accessing members such as id or objects on a model will trigger this, so it’s simplest just to disable the check.

output-format=parseable include-ids=yes

These makes the output of PyLint easier to parse by Buildbot, if you’re not using it then you probably don’t need to include these lines.

good-names= …,qs

Apart from a limited number of names PyLint tries to enforce a minimum size of three characters in a variable name. As qs is such a useful variable name for a QuerySet I force this be allowed as a good name.

max-line-length=160

The last change I make is to allow much longer lines. By default PyLint only allows 80 character long lines, but how many people have screens that narrow anymore? Even the argument that it allows you to have two files side by side doesn’t hold water in this age where multiple monitors for developers are the norm.

PyLint uses the exit code to indicate what errors occurred during the run. This confuses Buildbot which assumes that a non-zero return code means the program failed to run, even when using the PyLint buildstep. To work around this I use a simple management command to duplicate the pylint program’s functionality but that doesn’t let the return code propagate back to Builtbot.

from django.core.management.base import BaseCommand
from pylint import lint

class Command(BaseCommand):
    def handle(self, *args, **options):
        lint.Run(list(args + ("--rcfile=../pylint.cfg", )), exit=False)

Communicating With Stakeholders

2011-02-25T13:04:09+00:00

Last week I had a discussion with my boss about the best way to communicate with my stakeholders about the progress of any work that they have asked my team to do. The question basically became how much information should be communicated during a project. The requirements and delivery phases obviously require close communication, but what is appropriate while the developers are hard at working, churning out code? If, like me, you are required to work a number of disparate departments then the people in the departments may want to know what work is currently on your plate before they ask you to do something. What’s the best way to keep a status board up to date?

Traditionally we have had a Bugzilla installation which was used to store a record of almost every change we made. A subversion post commit hook allows us to link every commit back to a piece of work in Bugzilla. This works well for coding in an issue-driven-development style, but does result in Bugzilla sending a lot of emails. Many of which are completely irrelevant to people outside of IT. Indeed even people inside IT, but who aren’t directly linked to that piece of work, don’t need to be informed by email of every checkin.

Recently we have begun to experiment with FogBugz. While similar to Bugzilla it has a number of subtle differences. Firstly FogBugz is designed to be used in a helpdesk environment so it provides the ability to communicate both within the team and with external stakeholders from the same interface. This gives you the ability to communicate on two different levels, with all the communication still being tracked. The second difference is that FogBugz is not known amongst the people outside of IT. Not all stakeholder know about Bugzilla, but some do, and some can even search and reply using the webinterface. With FogBugz we’ll take this ability away as, at least initially, only a limited number of IT people will have access to the cases.

FogBugz had other project management advantages to Bugzilla. It allows you to create subcases to break down a piece of work into more easily implemented chucks. It also has advanced estimation capabilities to allow you to project how long a milestone will take to complete.

The crux of discussion is this: which is better, a known tool that provides complete access to status to those that want it, or a tool that enables those doing the work to plan better but prevents those outside from seeing how the work is progressing for themselves?

In my view developers should use the tools that they feel help them work the best. Looking at a list of tasks is probably not going to help someone who is not doing the work to understand whether the project is on track or not. However, it is important that stakeholders are kept informed of progress regularly so switching to a less open development model should not be used as an excuse to become more insular, quite the opposite in fact. Switching to a less open development model should force you to be more explicit and include status updates as part of your regular working schedule.

Unittesting QSyntaxHighlighter

2011-02-16T13:04:09+00:00

I’m using test driven development while building my pet project, DjangoDE. A key part of an IDE is the syntax highlighting of the code in the editor, so that’s one area where I’ve been trying to build up the test suite.

To test the syntax highlighter the obvious approach is to send the right events to write some code into the editor the check the colour of the text. Although the QT documentation is usually excellent, it doesn’t go into enough detail on the implementation of the syntax highlighting framework to enable you to query the colour of the text. In this post I’ll explain how the colour of text is stored, and how you can query it.

A syntax highlighting editor is normally implemented using a QPlainTextEdit widget. This object provides the user interface to the editor and manages the display of the text. The widget contains a QTextDocument instance, which stores the text. To add syntax highlighting you derive a class from QSyntaxHighlighter then instantiate it, passing the document instance as the parameter to the constructor. This is explained in detail in the syntax highlighter example.

The document stores the text as a sequence of QTextBlock objects. These store the text as well as the formatting information used to display it. You might think that you can just call QTextBlock::charFormat to get the colour of the text. Unfortunately it’s not that simple as the format returned by that call is the colour that you’ve explicitly set, not the syntax highlight colour.

Each QTextBlock is associated with a QTextLayout object that controls how the block is rendered. Each layout has a list of FormatRange objects, accessible using the additionalFormats method. It is this list that the QSyntaxHighlighter sets to specify the colour of the text.

Now we know where the colour information is stored, we can find out what colour a particular character will be. Firstly you need to find out which QTextBlock the text you want is. In a plain text document each line is represented by a separate block, so this is quite straightforward. You then get the list of FormatRanges and then iterate through, checking to see if the character you want is between format_range.start and format_range.start + format_range.length

For an example of this you can check out the test file from DjangoDE here

Government And Website Launches

2011-02-03T13:04:09+00:00

On Tuesday the UK Police service launched a new website, with much fanfair, that allows you to see the crimes that have happened near where you live. This is part of the Government’s commitment to make data much more freely available than it has been in the past.

People are nosy. There’s no way around the fact that we just love to see what our neighbours are doing and to investigate the area around us. Every time a website like the crime maps, a new census, or noise maps launches the same sequence of events happen. The are news stories on TV, on radio and online and then the website crashes. We end up with stories like this and people will, in general, never go back.

If you investigate where this new site is hosted you’ll find it’s on RackSpace, who offer a cloud hosting service so really there’s no reason for the site to go down. Although the site is interactive there is no user generated data so the site should be easily scalable.

The site cost £300,000 to develop, and although 75,000 hits a minute is a lot surely will a little bit of thought the site should have scaled fairly easily?

DjangoDE 0.1 Released

2011-01-24T13:04:09+00:00

I spend most of time at work building websites in Django. My editor of choice until now has been Kate with Chromium open on another screen. Most of my co-workers use VIM inside a PuTTY session to do their editing. With multicore machines with gigabytes of RAM, surely there’s a better way?

I investigated the current state of Django IDEs and came to conclusion that none of them are that great. Most are a plugin to a giant IDE that tries accommodate many different languages, so each feels like second best, or they are designed more for traditionally programming and not webdevelopment so they don’t integrate with Django’s in built server and don’t support editing Javascript, or provide a built in webbrowser. I also don’t want to have to pay for the editor, which limits my choice even more.

Having decided that none of the existing IDEs quite fit my requirements I did what any self respecting open source programmer with an itch would do, I headed to Google Code, created a new project and then got coding.

The project has now reached the stage where I feel I can make my first release, although this is a release that is very much in the spirit of the open source mantra, “release early, release often.” The key features are all in place, which I’ll talk about below, but they’re all extremely basic and it’s not ready to be used as a day-to-day editor. If you’re looking for a Visual Studio replacement, this is not it.

The source can be downloaded from the project site, or PyPI. The simplest method is just to run sudo easy_install djangode. You’ll need to have PyQt and of course Django installed. This first release also requires Linux, but being a Qt application that requirement will not exist for ever and it will eventually work on Windows (and hopefully Macs too).n DjangoDE uses the manage.py file to define the scope of the project. The “Project” menu bar lets you pick a manage.py file to load, or to create a new Django project.n The editor has a partial syntax highlighter for Python code implemented, and a quick access bar on the left of the window which lets you jump straight to the view implementing a particular url. It also has a tab for each application that is active in your project which lists the models.n When you open DjangoDE it’ll show you the Django project website in a tab. If you enter “/” into the address bar it’ll load the front page of your website.

Pressing F9 while viewing a source code file will create a breakpoint. If you then use the browser to view a page that passes that line the code will break and that point and the debug bar will appear. This shows you the variables in the current scope.

It’s very early days for this project, but these features will be fleshed out in the coming releases. I’d like to encourage people to post issues if they come across bugs or want features implemented. You can also contact me through Twitter as djangoide.

Can the entrance barrier ever be too low?

2011-01-19T11:00:00+00:00

Yesterday Google announced a new feature for Google Code’s Project Hosting. You can now edit files directly in your browser and commit them straight into the repository, or, if you don’t have commit privileges, attach your changes as a patch in the issue tracker.

If you’re trying to run a successful open source project then the key thing you want is more contributors. The more people adding to your project the better and more useful it will become, and the more likely it is to rise out of the swamp of forgotten, unused projects to become something that is well known and respected.

It’s often been said that to encourage interaction you need to lower the barrier so that people can contribute with little or no effort on their part. Initially open source projects are run by people who are scratching their own itches, and producing something that is useful to themselves. Google’s intention with this feature is clearly to allow someone to think “Project X” has a bug, I’ll just modified the code and send the developers a patch. The edit feature is very easy to find, with a prominent “Edit File” link at the top of the screen when you’re browsing the source code so Google have clearly succeeded in that respect.

My big concern here is that committing untested code to your repository is right up there at top of the list of things that programmers should never, ever, do. I like to think of myself as an expert Python programmer, but I’ll occasionally make simple mistakes like missing a comma or a bracket. It’s rare that anything beyond a trivially small change will work perfectly first time. Only by running the code do you pick up these and ensure that your code is at least partially working.

I’m all for making it easy to contribute, but does contributing a large number of untested changes really help anyone? I’m not so sure. Certainly this feature is brilliant for making changes to documentation where all you need to do is to read the file to know that the change is correct, but it seems a long way from best-practice for making code changes.

Perhaps I should be thinking about this as a useful tool for sketching out possible changes to code. If you treat it as the ability to make ‘pseudo-code’ changes to a file to demonstrate how you might tackle a problem it seems to make more sense, but open source has always lived by the mantra ‘if you want it fixed, fix it yourself’.

I suppose I should worry about getting my pet open source project to a state where people want to contribute changes of any quality, and then I can worry about making the changes better!

Rsync backups to Amazon S3

2011-01-14T11:00:00+00:00

Having recently got married I wanted to make sure all the photos taken at the event are safely stored for the posterity. I thought I’d take the opportunity of making sure that all the rest of my photos are safely backed up, and that any new ones are also backed up without me needing to do anything.

One of the simplest places to keep your backups is Amazon S3. There is essentially an unlimited amount of space available, and it’s pretty cheap. rsync is a great tool to use when backing up because it only copies files, and parts of files that have changed so it will reduce the amount of data transferred to the lowest amount of possible. With S3 you not only pay for the data stored, but also for the data transferred so rsync is perfect. So, how do we use rsync to transfer data to S3?

I won’t go through setting up an Amazon S3 or creating a bucket, the Amazon documentation does that just fine.

The first thing to do is download and install s3fs. This is a tool that uses FUSE to mount your S3 account as if it was an ordinary part of your filesystem. Once you’ve got it installed you need to configure it with your access and secret ids. You’ll be given these when you set up your S3 account. Create a file .passwd-s3fs in your home directory and chmod it so it has no group or other permissions.n Mounting your S3 bucket is simple, just run:

s3fs bucket_name /mount/point

Any file operations you conduct in /mount/point will now be mirrored to S3 automatically. Neat!

To copy the files across we need to run rsync.

rsync -av --delete /backup/directory /mount/point

This will copy all files from /backup/directory to /mount/point and so to S3. The -a option means archive mode, which sets the correct options for performing a backup. -v is verbose so you can see how far it gets while --delete means that files will be deleted from /mount/point if they’ve been deleted from the directory your backing up.

On your initial backup you’ll likely be transferring multiple gigabytes of data, and that will saturate the upload on your Internet connection. This prevents you from doing pretty much anything else until it’s finished, so lets look at limiting how fast the back up runs.

trickle is a very useful program that limits the bandwidth that a single program can consume. We don’t want to limit rsync, because that is running locally, it’s the s3fs program we want to limit so alter the mount command to be:

trickle -u 256 s3fs bucket_name /mount/point

This will only allow s3fs to consume a maximum of 256KB/s of upload, allowing you to continue to browse Facebook while the backup is progressing. Simply change to the upload number depending on how fast your internet connection to get the right balance between a usable connection and the speed of the backup.

To automate the backup just add the two commands to a script file and put it in your crontab like so.

@daily /home/username/bin/s3_backup

Using Python Logging Effectively

2011-01-07T11:00:00+00:00

If you’re writing a Python program that doesn’t have a text-based user interface (either it’s a GUI or runs as part of another program, e.g. a webserver) then you should avoid using the print statement. It’s tempting to use print to fill the console with information about what your program is up to. For code of any size though, this quickly devolves into a hard to navigate mess.

Python’s standard library contains a module, logging, that lets you write code to log as much information as you like and configure what you bits you are interested in at runtime.

There are two concepts that you need to understand with logging. Firstly there is the logging level. This is how you determine how important the message is. The levels range from debug as the least important, through info, warning, error, critical to exception, the most important. Secondly there is the logger. This allows you divide your messages into groups depending on the part of your code they relate to. For example, you might have a gui logger and a data logger.

The logging comes with a series of module level functions by each of the names of the logging levels. These make it quick and easy to log a message using the default logger.

logging.debug("Debug message")
logging.error("Error retrieving %s", url)

The second of these two lines has more than one argument. In this case the logging module will treat the first argument as a format string and the rest as arguments to the format, so that line equivalent to this one.

logging.error("Error retrieving %s" % (url, ))

If you try to treat the logging code like you would a print statement and write logging.error("Error retrieving", url) then you’ll get the following, very unhelpfui, error message.

Traceback (most recent call last):
 File "/usr/lib/python2.6/logging/__init__.py", line 776, in emit
   msg = self.format(record)
 File "/usr/lib/python2.6/logging/__init__.py", line 654, in format
   return fmt.format(record)
 File "/usr/lib/python2.6/logging/__init__.py", line 436, in format
   record.message = record.getMessage()
 File "/usr/lib/python2.6/logging/__init__.py", line 306, in getMessage
   msg = msg % self.args
TypeError: not all arguments converted during string formatting

Notice how this exception doesn’t tell you where the offending logging statement is in your code! Now you know the type of error that will cause this that will help in tracking the problem down, but there is more than can be done to help you find it. The logging library allows you to specify a global error handle, which combined with the print stack trace function will give you a much better error message.

import logging
import traceback
def handleError(self, record):
    traceback.print_stack()
logging.Handler.handleError = handleError

Loggers are created by calling logging.getLogger('loggername'). This returns an object with the same set of log level functions as the module, but which can be controlled independently. For example:

gui_log = logging.getLogger('gui')
gui_log.debug("created window")
data_log = logging.getLogger('gui')
data_log.debug("loaded file")

Where this comes in really handy is when you set the level of messages that you want to see independently for each logger. In the next code block we set the logging module so we’ll see lots of debugging messages from the GUI and only errors from the data layer. Although here we’re setting the levels directly in code, it’s not a big jump to make them configurable using a command line option.

gui_log.setLevel(logging.DEBUG)
data_log.setLevel(logging.ERROR)

The logging module also lets you configure how your messages are formatted, and to direct them to files rather than the console. Hopefully this short guide is useful, let me know in the comments!

Evolving The Blind Watchmaker

2010-09-20T11:00:00+00:00

Recently I’ve been reading the classic book by Richard Dawkins, The Blind Watchmaker. In it he begins by discussing how evolution can produce complex systems from only a few simple rules. He demonstrates this using a simple tree drawing algorithm in which a few ‘genes’ control aspects such as the number of branches and the branch angle. The trees are evolved solely through mutation of an initial tree, rather combing the ‘genes’ of two trees to produce a child, and introducing mutations in those children.

In reality evolution is driven by pressures from the environment on the genes and those that produce the fittest host will survive. As this is early in the book though Dawkins uses himself as the environment and manually picks the most visually appealing trees.

Although the book is essentially timeless as although new evidence is continually being found in favour of evolution, the general thrust remains true. The passages where he talks about his computer, however, have dated horribly (which is not surprising given it was first published in 1986!). In this post I’ll describe how to recreate the section where he describes evolving trees in Python so you can create your own trees on your pc.

As with the book our trees will be controlled by nine genes, each of which is an integer. Dawkins doesn’t state what the nine genes do as for his purposes that would confuse matters, but for us it’s vital. Fortunately figure three allows us to work out what genes one, five, seven and nine do for ourselves.

Horizontal scaling
Number of branches per level
Length of first branch
Scaling factor for length of subsequent branches
Vertical scaling
Angle of first branch
Angle of branching
Scaling factor for angle of branching
Levels of branching

The gene descriptions in bold are those that I deduced from the book, the others are ones I decided on myself. The first step in writing a program like this is to decide exactly how these genes will affect the drawn tree. To do this we create a series of functions, one for each gene, that converts the gene value into a value that can be used by the drawing code. These functions are given below.

horiz_scaling = lambda dna: (dna[0]+10.0)/10.0
branches = lambda dna: dna[1]
initial_length = lambda dna: dna[2] + 10
length_scaling = lambda dna: (dna[3]+10.0)/10.0
vert_scaling = lambda dna: (dna[4]+10.0)/10.0
initial_angle = lambda dna: dna[5]/10.0
initial_angle_of_branching = lambda dna: 1.0+dna[6]/5.0
change_in_angle_between_branches = lambda dna: dna[7]/5.0
max_levels = lambda dna: dna[8]

These functions are used by the draw_branch function which renders a single line, and recursively calls itself to draw the next level of branches.

def draw_branch(img, dna, level, start, angle, length, angle_between_all_branches):
    end = (start[0] + math.sin(angle) * length * horiz_scaling(dna),
              start[1] - math.cos(angle) * length * vert_scaling(dna))
    img.line(start + end, (0, 0, 0))
    if level >= max_levels(dna):
        return
    else:
        branch_angle = angle - angle_between_all_branches/2.0
        angle_between_branches =
            0 if branches(dna) == 0 else angle_between_all_branches/branches(dna)
        for i in range(branches(dna)+1):
            draw_branch(img, dna, level+1, end,
                branch_angle + angle_between_branches*i, length*length_scaling(dna),
                angle_between_all_branches + change_in_angle_between_branches(dna))

To start the drawing process off we need a function, draw_tree, which calls the branch drawing function with the initial values for the length of branch and angle between the subbranches.

def draw_tree(img, dna):
    draw_branch(img, dna, 0, (50, 70),
        initial_angle(dna), initial_length(dna), initial_angle_of_branching(dna))

Now we can draw a tree a we need to be able to generate the children of tree, which we do by picking a gene and either incrementing or decrementing it. A couple of genes make no sense if they are negative so they code prevents these from going below zero.

def evolve(dna):
    gene = random.choice(range(9))
    if (gene in [1, 8] and dna[gene] == 0) or random.random() < 0.5:
        dna[gene] += 1
    else:
        dna[gene] -= 1
    return dna

If we combine these functions with a simple TK-based interface, as shown below, we exactly the abilities described in The Blind Watchmaker book. Nine possible trees are displayed, when the user clicks on one nine new children are created and displayed.

Without more details about the original program it’s hard to recreate it exactly, but this program is a decent starting point. Happy evolving!

To run this program yourself you’ll need to download and install Python from python.org and the PIL image library. Next code the sourcecode below into a file called “darwkins_trees.py” and double click on it.

#!/usr/bin/env python
# Copyright <year> <copyright holder>. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without modification, are
# permitted provided that the following conditions are met:
#
#   1. Redistributions of source code must retain the above copyright notice, this list of
#      conditions and the following disclaimer.
#   2. Redistributions in binary form must reproduce the above copyright notice, this list
#      of conditions and the following disclaimer in the documentation and/or other materials
#      provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND ANY EXPRESS OR IMPLIED
# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# The views and conclusions contained in the software and documentation are those of the
# authors and should not be interpreted as representing official policies, either expressed
# or implied, of Andrew Wilkinson.
import math, os, random, sys
from Tkinter import *
import Image, ImageTk, ImageDraw
image_size = (100, 100)
samples = 9
horiz_scaling = lambda dna: (dna[0]+10.0)/10.0
branches = lambda dna: dna[1]
initial_length = lambda dna: dna[2] + 10
length_scaling = lambda dna: (dna[3]+10.0)/10.0
vert_scaling = lambda dna: (dna[4]+10.0)/10.0
initial_angle = lambda dna: dna[5]/10.0
initial_angle_of_branching = lambda dna: 1.0+dna[6]/5.0
change_in_angle_between_branches = lambda dna: dna[7]/5.0
max_levels = lambda dna: dna[8]
def draw_branch(img, dna, level, start, angle, length, angle_between_all_branches):
    end = (start[0] + math.sin(angle) * length * horiz_scaling(dna), start[1] - math.cos(angle) * length * vert_scaling(dna))
    img.line(start + end, (0, 0, 0))
    if level >= max_levels(dna):
        return
    else:
        branch_angle = angle - angle_between_all_branches/2.0
        angle_between_branches = 0 if branches(dna) == 0 else angle_between_all_branches/branches(dna)
        for i in range(branches(dna)+1):
            draw_branch(img, dna, level+1, end, branch_angle + angle_between_branches*i, length*length_scaling(dna), angle_between_all_branches + change_in_angle_between_branches(dna))
def draw_tree(img, dna):
    draw_branch(img, dna, 0, (50, 70), initial_angle(dna), initial_length(dna), initial_angle_of_branching(dna))
def evolve(dna):
    gene = random.choice(range(9))
    if (gene in [1, 8] and dna[gene] == 0) or random.random() < 0.5:
        dna[gene] += 1
    else:
        dna[gene] -= 1
    return dna
class Application(Frame):
    def __init__(self, master=None):
        Frame.__init__(self, master)
        self.dna = [0, 1, 0, 0, 0, 0, 0, 0, 1]
        self.grid()
        self.create_widgets()
        self.create_choices()
    def create_widgets(self):
        self.buttons = []
        for i in range(samples):
            button = Button(self)
            button["command"] = self.choose_tree(i)
            button.grid(row=i / 3, column=i % 3)
            self.buttons.append(button)
    def create_choices(self):
        self.choices = [evolve(self.dna[:]) for i in range(samples)]
        self.images = [Image.new("RGB", image_size, (255, 255, 255)) for _ in range(samples)]
        [draw_tree(ImageDraw.Draw(self.images[i]), self.choices[i]) for i in range(samples)]
        self.tkimages = [ImageTk.PhotoImage(image) for image in self.images]
        for i in range(samples):
            self.buttons[i]["image"] = self.tkimages[i]
    def choose_tree(self, i):
        def func():
            self.dna = self.choices[i]
            self.create_choices()
        return func
if __name__ == "__main__":
    root = Tk()
    app = Application(master=root)
    app.mainloop()
    root.destroy()

Simplifying Django dependencies with virtualenv

2010-08-24T12:23:49+00:00

virtualenv is a tool for simplifying dependency management in Python applications. As the name suggests, virtualenv creates a virtual environment which makes it easy to install Python packages without needing root privileges to do so.

To use the packages installed in a virtual environment you run the activate script in the bin directory of the virtual environment. This is fine when you’re working on the command line, but you don’t want to have to remember this step when running the debug server, and it’s hard to get that to work when the site is deployed under mod_wsgi.

To make things easier you can add the appropriate directory from the virtual environment to Python’s path as part of manage.py, or the appropriate fcgi or wsgi control script.

import os
import sys
import siten
root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
site.addsitedir(os.path.join(root_dir, 've/lib/python%i.%i/site-packages'
                % (sys.version_info[0], sys.version_info[1])))

Just add the code above to the top of your manage.py file and the ve virtual environment will always be activated when you run the script.

Making History by Stephen Fry Book Review

2010-07-08T12:35:10+00:00

This book poses an interesting question, what would happen if you could stop Adolf Hitler from being born? Ask anyone if they had the choice, would they take it and I imagine that almost everyone would say yes. When Michael, the lead character in this story has that choice he takes it with both hands. Unfortunately though, things do not go according to plan.

Stephen Fry is a British institution and a well known upper class intellectual. In this book he is clearly writing about what he knows as it focuses on a post-graduate student at Cambridge. The book is full of colour and detail and has an air of authenticity that draws you into the world effortlessly. There seems to be a significant amount of himself in the lead character and he writes as he talks. At first I found the overly intellectual mode of writing to be annoying and distracting from what the author was trying to say. Persevere though, and you’re rewarded with a charming, warm and funny story that will also make you think about history and the choices we make.

This book can easily be described a genre-defying as the time-travel premise means it clearly fits into science fiction. Don’t worry though, this book is not about spaceships or lasers as the two moments of time-travel are tiny compared to their consequences, and combined with flashbacks into the two world wars, the book fits into historical fiction as well. Is historical science fiction a genre? It is now.

This book was first published in 1996, and if you haven’t read in the last fifteen years then you should definitely go and hunt down copy. You’ll read it and imagine that it was written yesterday.

Wii DVD

2010-07-07T12:28:35+00:00

This weekend I brought Super Mario Galaxy 2 for the Wii. I greatly enjoyed the first one, and it’s one of the few games that me and my girlfriend can play together.

It’s a pretty easy game to pick up, but for this one Nintendo included a tutorial DVD. Two questions immediately spring to mind here. Firstly, what can you do with a DVD that you can’t do with an in-game, interactive, tutorial. Secondly, why is this a DVD that you can’t play on the Wii?

Let me just say that again.

Why is this a DVD that you can’t play on the Wii?

To watch the tutorial you need to turn off your Wii, turn on your DVD player, watch the DVD, and then go back to your Wii. I’d love to know how Nintendo came to this decision, because it drastically slow you down your first experience of playing the game, and those that are mostly likely to want to watch the video are those who will struggle to switch over to the DVD player.

Come on Nintendo, you can do better than this.

Continuous Integration Testing

2010-06-30T13:16:27+00:00

At work we recently set up Buildbot to replace an in-house continuous integration tool that never really took off. I’ve used Buildbot before, in a hobby capacity, but using it my day job has really brought home to me how important not only testing is, but testing continuously.

Buildbot is pretty simple to explain. Every time you make a commit to your source control system this triggers buildbot to run a series of steps over your code. What these steps are is totally configurable by you. Typically though, you’ll configure it to check out your code, compile it and run unit tests.

I’m a big fan of test driven and issue driven development. With these two methodologies when you want to make a change you record the issue in your bug tracking software, then write a test that detects the deficiency in your code. Once you’ve done that you can start actually making the change that you wanted to make in the first place. By writing down the issue you’re forced into thinking about the change that you want to make, and creating the test ensures that you know when it’s complete.

TDD and IDD are useful things to aim for, but in reality you’ll probably fall short sometimes. Not everything can be unit tested easily, or you might be under significant time pressures that force you to cut corners. One of the projects I manage has a test suite that takes 25 minutes to run. If I’m getting phone calls urging me to fix a problem as soon as possible then I can’t wait for the tests to run before I commit the change.

Because builtbot runs without user intervention you don’t need to remember to do anything. Buildbot is also a nag. If you introduce a bug with a change you quickly get an email letting you know that there is a problem, and if you set those emails up to go to your team then you also have peer pressure to get that problem fixed.

I spend most of my time building Django based sites. For these sites I set up six steps for Buildbot to follow. The first step is the checkout step, then a step to copy a settings file into place. Django stores settings in a Python file and these settings need to be configured for each server that the site is run on. I typically have a file settings_local.py.buildbot stored in the repository and this step copies this file to settings_local.py so the following commands will run with the correct settings. The next step is to download and install the dependencies of the site. I’m going to go into more detail on how I manage settings and dependencies in a future post, but Buildbot just runs the correct commands.

The first three steps that Buildbot runs are just configuration steps. The first of the real test is a simple compilation step. It’s rare that a change gets committed without it ever being run, but it’s still worthwhile to check that the code compiles. It is possible that when merging changes from different branches errors can creep in. Fortunately Python makes it easy to check that all the files in a directory structure compile using the compileall module. This can be run using the command python -m compileall -q which return an exit code of zero if all the files compile, and nonzero if there is an error. This means that if there is an error Buildbot will detect it, and mark the build as failed.

The next step I run is the unit tests. This where the bulk of the testing occurs. I’m not going to go into details on how to write unit tests, but assuming you have them this assures you that your tests pass after every commit. Because Buildbot can be easily configured to build on several different machines on each commit you can easily test that a change you made on a Python 2.5 machine also works on a Python 2.4 machine.

The final step I run is PyLint. PyLint checks that your code complies with a set of coding standards and simple checks. If your unit tests don’t have a 100% coverage (and frankly, whose does?) then PyLint will hep detect that there are no obvious flaws in your code. PyLint takes a bit of time to configure, especially as it doesn’t like some of the metaclass magic that Django uses, but once you have a configuration file written PyLint is an excellent final check.

Hopefully this has given you a flavour of what Buildbot can do for you, and if you’re in a team or are building a cross-platform piece of software then Buildbot really needs to be there, watching your back.

Why Does e=mc2? Book Review

2010-06-24T12:20:19+00:00

Rock star turned physicist Brian Cox and Jeff Forshaw have written a book with a deceptively simple title, Why Does e=mc² ?. With a title like that you might expect that the book will be along the same lines as the New Scientist books How To Make A Tornado and Do Polar Bears Get Lonely?. Instead though, you get a book that takes you on a in-depth journey through deriving the equation from first principles* and on to the many things that it implies.

As you might expect for a book from two physicists it is a fairly specific book, focusing on entirely on the equation and the mathematics and physics that surround it. If you’re looking for a history of the equation and the Einstein then this is not the book for you. That’s not to say that it’s not well written, or not accessible because it is both of those things. The jovial nature of the writing and understandable metaphors really help you to follow the progress of the book, especially when four dimensional space-time is being discussed.

This book is well written, informative and entertaining. If you’ve ever wanted to know why e=mc² and you’ve got a basic understanding of maths then “Why Does e=mc² ?” will answer all your questions and more.

Well, from an fairly basic set of set of assumptions.

Using Fabric For Deployment

2010-06-16T12:18:02+00:00

In a previous post I discussed what you want from an automatic deployment system. In this post I’ll discuss how use to solve the repeatability and scalability requirements that I set out.

Fabric is a tool which lets you write scripts to automate repetitive tasks. So far, so bash like. What sets Fabric apart is the tools it gives you to run commands on remote servers. Fabric allows you to run the same commands on multiple machines, and to move files between the hosts easily.

To get started with Fabric you’ll need to install it, but a simple sudo easy_install fabric should be enough to get you up and running. The Fabric website has excellent documentation, including a tutorial, but before I discuss how to integrate Fabric with your Django deployment process, lets go over the basics.

A Fabric control file is a Python file named fabfile.py. In it, you define a series functions, one for each command that you want to run on the remote servers.

from fabric.context_managers import cd
from fabric.operations import sudon
env.hosts = ['host1', 'host2']
def update():
    with cd('/data/site'):
        sudo('svn up')
        sudo('/etc/init.d/apache2 graceful')

We’ve defined the function update which can be run by typing fab update in the same directory as fabfile.py. When run, Fabric will connect in turn to host1 and host2 and run svn up in /data/site and then restart Apache.

Typically I define two functions. An update command, like that above is used to update the site where is has previously been deployed. A deploy command is used to checkout the site onto a new machine. Fabric lets you override the host list on the command line using the -H option. To deploy one of my sites on a new box I just have to type fab -H new-machine deploy and the box is set up for me.

Fabric helps you fulfil a few of the requirements for a perfect deployment system. It is scaleable, as to extend your system to a new machine you only need to add the hostname to the env.hosts list. It is also repeatable, providing you put every command you need to run to update your site into your fabfile.

With an automated deployment system in place we can now move on to looking a dependency, settings and database change management, but those are subjects for a future post.

iLust

2010-06-02T12:25:55+00:00

This weekend, the day after the iPad was released in the UK, I went into an Apple store and had a play with an iPad for a few minutes. My first reaction was surprise was that I was able to get access to an iPad so quickly. When I tried the same for the iPhone I had to give up as it was taking too long to queue. That’s probably down to the time of day rather than the popularity of the device though.

Much has been made of the iPad being a jumped up iPod Touch, and when the iPad was announced I was disappointed that it didn’t run Mac OS X. Having played with the device though I’m inclined to think that it was a good move. When they release the upcoming version 4 operating system with its background processing capabilities most criticisms will no longer be valid.

The iPad is usually described as a content consumption device, rather than a content creation device. That’s not quite true, as I could quite happily type this blog post using an iPad. However, as a webdeveloper in both a professional and a hobbyist capacity it is impossible to imagine using the iPad to create anything that complicated.

At the moment I have a large, dual screen, pc which is on most of the time while I’m at home. I might be sat on the sofa watching the TV, but my pc is on an whirring quite loudly a short distance away. The reason it’s on is so that I can check facebook, my email or google reader without having to wait a couple of minutes for it to switch on. It is exactly this sort use that the iPad will excel at. It is also the what the vast majority of computers are used for.

With an iPad my main computer could be off unless I want to actually do something constructive with my pc. As the iPad turns on instantly and has a long battery life it can live on my coffee table and be accessed as and when it’s needed. This is greener and will save me money on my electricity bill, although just how long it would take me to save enough money to claw back the purchase price I’m not sure. You’re probably talking on the order of decades though!

I’m an almost complete cloud computing convert. I use GMail for my email. I read my RSS feeds using Google Reader and create documents and spreadsheets using Google Docs, and I store my photos online using Flickr. All bar the last of these are supported easily by the iPad. The inability to plugin in my camera and upload photos to Flickr means that even for my parents, their current computer won’t be going anywhere.

If this blog post sounds like someone who is trying to convince themselves to buy an iPad, then you’re probably right. Unfortunately my upcoming wedding means that I’ve had to put my planned iPhone 4 purchase this summer on hold. I’ll just have to tell myself that the iPad 2 will be even better…

Perfect Deployment Of Websites

2010-04-15T12:17:32+00:00

Recently I have been giving a lot of thought to how best to deploy websites, specifically Django powered sites. In future posts I’ll describe how I use some of tools available to deploy websites, but in this post I want to set out the goals of any system that you use to deploy a website.

What do mean when we say deployment? Clearly it involves getting your code onto a production server but the process also needs to look after any dependencies of your code. Updates also sometimes require database changes and these need to be managed and deployed with the appropriate code changes. If your website is more than just a hobby then it will also usually involve some sort of high availably set up.

The first requirement is repeatability. You might be able to follow a list of ten commands without making a mistake normally, when your site is broken and you need to get a fix deployment as soon as possible following that list will suddenly become a whole lot harder. For this reason, and to avoid the temptation to cut corners when deploying a change automation of as much as possible is key.

The second requirement is scalability. As your website grows you deployment process needs to be able grow with it. As you add a new server to your cluster you don’t want to have to spend a long time updating your deployment process, nor do you want the extra server to create extra work for you to do on every deployment.

Another requirement is speed. I’m usually very skeptical of anything that claims ‘written for speed’ as a key benefit. Unless you know something is slow, it’s usually better to make something easier to maintain than quick. An automatic process will inevitably be quicker than a manual one and whether or not your deployment process results in down time for your site the upgrade process is inevitably a risk and ideally that risk window will be kept to a minimum.

Database migrations are a tricky thing to get right. A deployment system must allow developers to track changes that they make to the database, and to make it easy to ensure these changes are applied at the right moment.

In future posts I will talk about how tools such as fabric, virtualenv, pip and south can be used to meet these requirements and ensure you never have a failed deployment again.

XBox 360 USB Hard Disks

2010-03-19T13:24:12+00:00

Today some very strong rumours have appeared which state that Microsoft is planning to introduce the ability to use an external USB hard disk with the XBox 360. Bizarrely these are going to limited to 16GB, which for a modern harddisk is incredibly tiny. The Xbox has recently been updated to include a 250GB hard disk in the top of the range version but, that is still puny in hard disk terms. Given that more and more games feature DLC, and Microsoft is pushing the movie watching capabilities these limits on the amount of data you can store can easily bite even moderate users, let alone hard core gamers.

I do love my XBox, but it’s the artificial limitations like this that really annoy me.

Championship Manager 2010 for the iPhone Review

2010-03-17T13:20:07+00:00

I often pass some time while traveling playing a game or two on my phone. Being a football fan I was interested to see that a version of Championship Manager was available for the iPhone.

Early impressions of the game are good. The interface is polished with plenty of options and buttons to press. The game also features a pretty complete set of teams and players with all the usual statistics that you would expect from a football management game.

The key part of a football management is the match interface. Choosing your formation and assigning players to positions is incredibly easy with the touch interface. The actual view of the match is top down and feels really cramped on the small screen. Reading the text commentary is simple, but watching the tiny dots running around on the pitch is very difficult to follow. You cannot help but feel detached from the action.

Post-match you are often treated to a press conference where you pick a journalist who asks you a question. You then pick an answer from a number of options, trying to balance the happiness of your players, board, fans and the media. Different answers will improve your standing with some and reduce it with others. Oddly along with the text of each answer the game shows you the exact effect the answer will have. A really manager has to guess at how their answer will be interpreted, they don’t have a help set of red and green numbers.

I was quite surprised to see a football management game on the iPhone. Both Championship and Football Manager are well known for needing substantial amounts of CPU power and memory to run well even on a desktop PC. The game does run pretty well, although some of the menu transitions are a little sluggish. I’m sure it would run better on a 3GS rather than the 3G that I have. Another issue is that the game is a huge battery drain and you can probably only get an hour or so play on a fully charged phone.

While Championship Manager on the iPhone was never going to be a graphical marvel, but it is undoubtedly one of the most impressive games on the system. At just £2.99 it’s an excellent purchase.

Save BBC 6 Music

2010-03-08T13:28:10+00:00

I’ve been a regular listener to 6music since about six months after it launched. Below is the email I’ve sent to the public consultation on the future of the station, as it has been proposed that the station be closed to save money which can be spent on making other programs.

Dear Sir/Madam,

I have been a regular listener to BBC 6 Music since about six months after it launched. Apart from the Today Program and a few comedy shows on Radio 4 it quickly became the only radio station I listened to. It provided the sound track to much of my University life and and introduced me to a wide range of new music that I would have never found otherwise. It has been tremendously influential and substantially shaped my taste in music.

I believe that closing 6 Music will prove to be a huge mistake and a massive step backwards for the BBC. If the BBC is to stand for excellence then closing a radio station that is staffed by people at the forefront of music seems to go against that. 6 Music’s track record of finding a promoting new bands is something that BBC should be extremely proud of.

I believe that neither the BBC nor commercial stations offer a suitable replacement for 6 Music. Although 6 Music is associated with Radio 2 in style they are widely different. Although some assurances have been given that the best of 6 Music will be relocated to other stations I cannot see how this can be done during the peak listening hours without alienating the listeners of Radio 1 and 2. 6 Music is by it’s very nature targets a niche audience, but its influence stretches far beyond its reach.

The fact that 6 Music is only available on DAB rather than FM radio is a key factor in its limited audience reach. It is probably the most well known of all the digital-only stations and key driver in transitioning people away from the antiquated analogue radio infrastructure. I own three digital radios which enable me to listen to 6 Music throughout my house. DAB certainly has its critics and loosing such a jewel in DAB radio’s content reduces rather than increases the incentive to go digital.

I strongly believe that 6 Music is excellent value for money and I urge the BBC Trust to reject the recommendation to close it.

Yours Sincerely, Andrew Wilkinson

Feel free to use my letter for inspiration, but please don’t copy it wholesale. That will not help to save the station that we love. Find details on how to response here.

Naming Screen Sessions

2010-02-03T13:27:53+00:00

I develop a number of Django-powered websites at work, and usually I want to leave them running when I’m not working on them so others can check out my progress and give me suggestions. The Django development server is incredibly useful when developing, but it’s not detached from the terminal so as soon as you log out the server gets switched off. One alternative is to run the website under Apache, as you would deploy it normally. This solves the problem of leaving the website running, but makes it much harder to develop with.

A third option is the GNU program Screen. When run without arguments screen puts you into a new bash session. Pressing Ctrl+d drops you back out to where you were. The magic occurs when you press Ctrl+a d. This drops you back out, but the bash session is stilling running! By typing screen -r you’ll reattach to the session and can carry on working as before. You can leave it as long as you like between detaching and reattaching to a session, as long as the computer is still running.

It is possible to run multiple screen sessions at once, perhaps with a different Django development server running in each. Unfortunately screen will only reattach automatically when there is just one detached session. If you have more than one then you’ll be confronted by a cryptic series of numbers that uniquely identifies each session. You can reattach to a specific session you can type screen -r <pid>.n To make things easier to reattach to the session that I’m working on I give these sessions name so rather than a cryptic series of numbers I see a useful set of names. To do this you just need to type Ctrl + A : sessionname name.

There are plenty of other useful things that screen can do, but named sessions is by far and away the most common one that I use.

Where GitHub (Possibly) Went Wrong

2010-01-27T13:21:33+00:00

While on my delayed train this morning I was listening to episode 80 of the excellent Stack Overflow podcast. In this episode Jeff Atwood was complaining to Joel Spolsky about his problems with GitHub.

GitHub is a social coding site, along the same lines as Sourceforge or Google Code, but focused entirely on the distributed version control system Git. Where GitHub differs from the other project hosting sites, and where I think Jeff’s confusion comes from is that with GitHub the primary structure on their site is that of the developer, not of the project. They treat every developer as a rock star, who is bigger than the projects that they work on.

GitHub makes it incredibly easy to take a codebase, make your own changes and to publish them to world. What GitHub fails to do is to encourage people to collaborate together to push one code base forward. What I’m not suggestion is that branching is a bad idea. Branching code is a useful coding technique which can be used to separate in-development features from other changes until the code has stabilised again. What GitHub focuses on is the changes that an individual developer makes, not the changes required for a particular feature.

When a developer creates a copy of some code of GitHub they get a wiki and an issue tracker as well. This further confuses matters because not only do you have trouble knowing which git tree is the correct one to pull from, but you also don’t know where to report bugs or go to for documentation.

Google Code seems to be in a better position for combining distributed version control with project management. They have an excellent wiki and issue tracker, and give each project a straightforward and simple homepage. You can also use Mercurial, which is similar to git, as your version control system. All that they need to do is allow developers to publish their own changes, but in a markedly separate section to the core code of the project.

I can see how GitHub is nice for developers, but in any mildly successful open source project the number of users vastly outweighs the number of developers. It seems crazy to me to make your primary web presence suited only for the minority of people who are involved with the project.

Searching Stemmed Fields With Whoosh

2010-01-21T13:28:16+00:00

Whoosh is quite a nice pure-python full text search engine. While it is still being actively developed and is suitable for production usage there are still some rough edges. One problem that stumped me for a while was searching stemmed fields.

Stemming is where you take the endings off words, such as ‘ings’ on the word endings. This reduces the accuracy of searches but greatly increases the chances of users finding something related to what they were looking for.n To create a stemmed field you need to tell Whoosh to use the StemmingAnalyzer, as shown in the schema definition below.

from whoosh.analysis import StemmingAnalyzer
from whoosh.fields import Schema, TEXT, ID
schema = Schema(id=ID(stored=True, unique=True),
                       text=TEXT(analyzer=StemmingAnalyzer()))

Using the StemmingAnalyzer will cause Whoosh to stem every word before it is added to the index. If you use the shortcut search function to search with a word that should be stemmed it will return no results, as that word does not exist in the index, even though it was included in the data that was indexed.

To correctly search a stemmed index you must parse the query and tell the parse to use the Variations term class. The causes the words in the query to also be stemmed, so they correctly match words in the stemmed index.

searcher = ix.searcher()
qp = QueryParser("text", schema=schema, termclass=Variations)
parsed = qp.parse(query)
docs = searcher.search(parsed)

Custom Podcasts With MythTV

2009-12-17T13:31:03+00:00

I love listening to both BBC Radio 4 and BBC 6 Music. Like the rest of the BBC radio stations a significant proportion of the shows are available as a podcast. Unfortunately this is not true of all the shows, and for those that feature music such as Adam & Joe or Steve Lamacq the podcasts are talking only.

I watch almost all of TV through MythTV which records all of my favourite shows automatically while on my way to work I like to listen to podcasts that are downloaded automatically by iTunes. Would it be possible to automatically record shows with MythTV that aren’t available as podcasts and sync them to my iPhone automatically?

Recording a radio show with MythTV is no different to recording a TV show so that’s not a problem. MythTV also provides the ability to run a script after certain shows have been recorded. All that is required is a script that converts the recording into an mp3 file and to build an RSS feed which can be read by iTunes.

First we need to convert the recorded file into an mp3, which is easy to do with the ffmpeg program.

#!/usr/bin/python
# -*- coding: utf-8 -*-

from datetime import date, datetime
import glob
import MySQLdb
import os
import sys

input = sys.argv[1]
input_filename = input.split("/")[-1]
output_filename = input_filename.split(".")[0] + ".mp3"

os.system("ffmpeg -y -i %s -acodec libmp3lame -ab 128k /var/www/localhost/htdocs/podcasts/%s > /dev/null" % (input, output_filename))

Next up we need write out the RSS feed that iTunes will read. We start off by opening the file and writing out some boiler plate code.

fp = open("/var/www/localhost/htdocs/podcasts/feed.rss", "w")
fp.write("""<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" version="2.0">
  <channel>
       <title>MythTV Recorded Radio</title>
       <description>Radio Recorded By MythTV</description>
       <link>http://192.168.0.8/podcasts/</link>
       <language>en-us</language>
       <lastBuildDate>%(datetime)s</lastBuildDate>
       <pubDate>%(datetime)s</pubDate>
       <webMaster>andrewjwilkinson@gmail.com</webMaster>

       <itunes:image href="http://192.168.0.8/podcasts/stevelamacq.jpg"/>

       <itunes:category text="Technology">
           <itunes:category text="Podcasting"/>
       </itunes:category>
""" % { "datetime": datetime.now().ctime() })

Finally we need to write out a small bit of XML for each file that’s in our directory waiting to be downloaded. We do this by looking at each mp3 file in the podcasts directory and looking for the appropriate entry in MythTV’s recorded table. If an entry doesn’t exist then recording has been deleted and we delete the mp3 file.

db = MySQLdb.connect(user="mythtv", passwd="mythtv", db="mythconverg")

for radio_file in glob.glob("/var/www/localhost/htdocs/podcasts/*.mp3"):
    output = radio_file.split("/")[-1]
    size = len(open(radio_file, "rb").read())

    c = db.cursor()
    c.execute("SELECT title, description, starttime FROM recorded WHERE basename=%s", (output.split(".")[0] + ".mpg", ))
    row = c.fetchone()
    if row is None:
        os.remove(radio_file)
        continue

    title, description, starttime = row

    fp.write("""       <item>
           <title>%(title)s - %(datetime)s</title>
           <link>http://192.168.0.8/podcasts/%(output)s</link>
           <guid>http://192.168.0.8/podcasts/%(output)s</guid>
           <description>%(description)s</description>
           <enclosure url="http://192.168.0.8/podcasts/%(output)s" length="%(output_size)s" type="audio/mpeg"/>
           <category>Podcasts</category>
           <pubDate>%(datetime)s</pubDate>
       </item>""" % { "title": title, "description": description, "datetime": starttime, "output": output, "output_size": size })

fp.write("""
    </channel>
</rss>
""")

To use this put all three bits of code into one file, save it somewhere and mark it as executable. Next set up Apache to serve the directory /var/www/localhost/htdocs/podcasts/ as /podcasts. Finally you need to set up the script to run automatically after a program you want to create a podcast from has been recorded. To do this run mythtv-setup and select the ‘general’ menu option. Move through the screens until you reach ‘Job Queue (Job Commands)’. Add a brief description of the script in the ‘description’ field then enter the <path to script> %s. Then use the normal MythTV frontend and edit the recording schedules to make the correct User Script run.

Point iTunes at http://your.ip.address/podcasts/feed.rss and it’ll automatically download any new recordings.

CouchDB Document Cache

2009-09-29T11:28:41+00:00

It’s well known that one of the best things you can do to speed up CouchDB is to use bulk inserts to add or update many documents at one time.

Bulk updates are easy to use if you’re just blindly inserting documents into the database because you can just maintain a list of documents. However, a common scheme that I often use is to call a view to determine whether a document representing an object exists, update it if it does, add a new document if it doesn’t. To help make this easier I use the DocCache class given below.

The cache contains two interesting methods, get and update. Rather than writing directly to CouchDB when you want to add or update a document just pass the document to update. This will cache the document and periodically save them in a bulk update.

It is possible that you will retrieve a document from CouchDB that an updated version exists in the cache. To avoid the possibility that changes get lost you should pass the retrieved document to get. This will either return the document you passed in or the document that’s waiting to be saved if it exists in the cache. Because there is a gap between when you ask for document to be saved and when it actually is saved any views you use may be out of date, but that’s the cost of faster updates with CouchDB.

One complicating factor in the code is that the updating process updates the documents you passed in with _id and _rev from the newly saved documents. This means you can cache documents in a your own datastructure and should you decide to save the document again you won’t get a conflict error because it will have been updated for you.

class DocCache:
    def __init__(self, db, limit=1000):
        self._db = db
        self._cache = {}
        self._new = []
        self._limit = limit
        self.inserted = 0

    def __del__(self):
        self.save()

    def get(self, doc):
        if "_id" in doc and doc["_id"] in self._cache:
            return self._cache[doc["_id"]]
        else:
            return doc

    def update(self, doc, force_save=False):
        if "_id" in doc:
            self._cache[doc["_id"]] = doc
        else:
            self._new.append(doc)
        if force_save or len(self._cache) + len(self._new) > self._limit:
            self.save()

    def save(self):
        docs = self._cache.values() + self._new
        if len(docs) > 0:
            inserted_docs = self._db.update(docs)
            for doc, newdoc in zip(docs, inserted_docs):
                if newdoc[0]:
                    doc["_id"], doc["_rev"] = newdoc[1], newdoc[2]
                    self.inserted += 1
            self._cache = {}
            self._new = []

Charming Roulette

2009-09-25T11:28:25+00:00

Recently I went to a wedding which had a casino theme. To keep the guests entertained they gave every guest $100 from the Bank Of Fun to spend on the roulette and black jack tables. I decided to play roulette and I knew that the best way to maximise my chances of winning was to bet only on odd or even and to double my bet whenever I lost. At one point I was 2.6x up on my initial stake, but unfortunately, as you’d expect, I eventually lost the lot.

I want to see what I could have done to increase my peak winnings, and to try my best to leave the table with a positive cash flow. To do this we’ll simulate a roulette table using Python and try out various betting strategies. The Roulette wheel that was used at the Wedding was an American wheel and featured the numbers 1 to 36 as well as 0 and 00. Betting on odd or even will win if a number 1 to 36 comes up and it is odd or even. 0 or 00 will lose you your money. If you win your stake is doubled. This means that by betting on odd or even you stand a 47% chance of winning.

To help work out the best strategy we need to build a roulette wheel simulator. To do this we use the Python function given below. It takes four parameters and returns the amount money left at the end of the run. The first parameter is the amount of money to start with, the second is a function which takes the current amount of money and returns the bet. The next to function determine when to give up - either a limit on the number of rounds, or the amount of money to stop at. The variable wheel is a list containing 18 “odd” strings, 18 “even” strings as well as one “0” and one “00” string.

from random import choice
def roulette(stake, bet_func, go_limit=None, walk_away=None):
    go = 0
    while stake > 0 and (go_limit is None or go < go_limit) and (walk_away is None or stake < walk_away):
        go += 1
        bet = bet_func(stake)
        if bet > stake: bet = stake
        if choice(wheel) == "odd":
            stake += bet
        else:
            stake -= bet
    return stake

So, with the simulation in place let’s start working out some odds. The simplest betting strategy is to bet $1 each round. To do this we used this simple betting function.

def flat_bet(stake):
    return 1

The graph below shows how likely you are to win when following this strategy for the given target. As you can see if you only want to increase your money from $100 to $101 then you’ve a 90% chance of doing this betting $1 each go. However, if you set your sights higher then your chances quickly diminish and you’ve almost no chance of making even a $40 profit.

The strategy I used was to double my bet every time I lost and reset to a $1 bet when I won. This means that on average you only stand to win $1 per round, but because your bet is doubled each win wipes out any previous loses. The code for this bet function is more complicated and we need to use a callable class to store the state of our bet.

class scale_bet:
    def __init__(self, scale):
        self.bet = 1
        self.scale = scale
        self.prev_stake = None
    def __call__(self, stake):
        if self.prev_stake is None or stake > self.prev_stake:
            self.bet = 1
        else:
            self.bet *= self.scale
        self.prev_stake = stake
        return math.floor(self.bet)

The probably of winning is much better with the doubling strategy, and if you’re aiming for increasing your cash pile to $250 then you have a 25% chance of doing that.

The chances of winning are much better if you double your bet, but why stop at doubling? In the next test I aimed for a target of $200 and increased the scaling factor of the bet from 0.1 to 50. You can see from the graph below that increasing the scaling factor doesn’t change your chances of winning, instead it remains at about 47%.

The final chart shows the chance of reaching $200 with a bet which doubles when you lose. In this test the starting bet is set so that you have at least x goes remaining. We begin with having only one possible other bet, and go up to twenty. Despite what you might think, the chances of winning do not really change much.

So, what’s the outcome of all this? What ever you do, you’ve got a less than 50/50 chance of winning, but doubling your bet each time you lose will give a longer run before your lose your house.

Testing A Facebook Connect Site

2009-07-02T11:17:59+00:00

I’ve been developing a website in my spare time. Because I want to add plenty of social features it makes sense to let users login using Facebook Connect. The Facebook platform is by far the most successful social platform with many developers having created applications and websites that use it. I expected that the experience for developers would be a good one. Unfortunately, I was disappointed.

Facebook makes it easy to register an application and provide links to libraries that wrap their API and make it easy to get started. What Facebook don’t provide however is a downloadable version of their API to test with. Facebook have made some effort to support test users, but you have to open ports in your firewall and use your real facebook account to test with. Testing a new user signing up for your app is really quite a chore. Automating this sort of test is essentially impossible.

In an ideal world Facebook would produce a downloadable program that you can use to automatically user, programmatically log in users and generally automatically test all the parts of your code. The danger is that they’d have to give you a downloadable copy of their website code. Google App Engine give a similar downloadable environment, and you can’t say that Google don’t have a load of code that they don’t want to give away!

The Facebook API is pretty simple to get started with, and with in a couple of minutes you’ll have the code written to log a user in. Checking that it all works though, is a much tougher challenge…

Interestingness

2009-06-18T11:13:53+00:00

As an amateur photographer I upload all my photographs to Flickr. Most of the them are mediocre, but one or two are good enough that I think they can stand along side the photos from more professional users of Flickr.

For the same reason that I blog, I put my photos on Flickr because I feel that I have something useful or interesting to offer and to interact with new and interesting people. My blog gets between twenty and thirty visits a day - not much, but roughly the same as the number of visit I get to my photos on Flickr. The difference is that I only have twenty posts on my blog, whereas I have 2,000 photos on Flickr!

Plenty has been written about search engine optimisation for blogs, but not much has been written about SEO for Flickr. The majority of my photos have five or so tags, a title and are geotagged. Flickr does allow you to write a description and this would increase the about of text thereby giving search engines much more to go on. The key to gaining exposure on Flickr though, is to appear on Explore.

Flickr are not explicit about whether photos that appear in Explore are influenced by humans or not. They certainly imply that it’s chosen algorithmically though. If it’s chosen by computer then it should be possible to help your photos gain more exposure, beyond just taking nice photographs. If you look at the people who have their photos on Explore two things just out at you. Firstly it’s that they have a lot of contacts, and secondly that all their photos have lots of comments. You’d expect photos that appear on Explore to have a lot of comments, but typically all their photos have lots of comments. This implies two things, that you need to be active in the Flickr community, and that your contacts need to be active in looking at and commenting on your photos.

It appears that Flickr’s definition of Interestingness rewards not only excellent photos but also active community members. This is a really excellent design decision on Flickr’s part because it almost completely removes the ability to ‘spam’ Explore - you do have to be active and to be producing great photos to get features.

So, how do you get your photo featured on Explore? Well, you need to be taking great photos, submitting them to groups and interacting with other users. Like the best photos, it’s hard work, with a touch of luck.

WWDC 09

2009-06-11T12:08:21+00:00

It was with some trepidation that I listened in to this Monday’s Apple developer event, the WWDC keynote address. I have 16GB iPhone 3G, a current top-of-the-range model. With all speculation before the event it was clear that Apple were going to release a new model. But what were they going to include? Were they going to include the kitchen sink as some had been suggesting?

Fortunately, as the change in name would suggest, the new iPhone 3GS is an evolution rather than a revolution. Apple claim it has twice the magic which should equate to much faster application loading and probably better games too. In reality it’ll mean twice the cpu speed or twice the memory, or more likely both. It appears that the biggest change is that the iPhone 3GS contains a new graphics chip which gives it seven times the graphics throughput, Seven times!

The extra disk space that comes with a 32GB 3GS is nice, but is unlikely to be a reason to pay the extra for a 3GS. The same with voice dialling. The new phone does contain a compress, which will certainly make using the mapping application easier, and will allow for some really nice apps. When Google change Google Earth to us the compress it’ll be really nice to use.

I’m not going to pay the extra to upgrade before my contract is up, but I’ll certainly be a bit jealous of those with a new 3GS.

FlightControl Review

2009-06-01T11:33:07+00:00

On Friday I download a fun little puzzle game for my iPhone, FlightControl.

The premise of the game is that you’re running air traffic approach control for a small airport and you need to arrange for the two types of passenger jets, light aircraft and helicopters to land in the appropriate places without crashing into each other. A simple concept with even simpler controls. You tap on the plane you want to direct and then drag the plane to the runway. It will then follow the path you dragged out. It’s incredibly easy to use and really lets you focus on the goal of stopping those planes from crashing.

The graphics and sounds are excellent. The game has a great cartoon feel and although the menu and ui are minimal it has a very consistent look that clearly didn’t happen by accident. The map and airport look good and there are plans to add more airports to the game which I hope will be done to a similarly high standard.

The game starts off very easy to let you get the feel for the controls but the difficulty level ramps up pretty quickly and you’ll soon have to deal with five or more planes at once. When you’ve got two planes flying at different speeds trying to land on the same runway your brain will start to melt, but in a good way.

The game features online leaderboards which is a nice touch, but like with most online stats the leaders are way out of most users reach. The current all-time top score is almost 15,000. My best is 53.

My only criticisms are that the airport is perhaps a little large which means you don’t have much room to sort your planes into stacks as you wait for them to land. The game also has an annoying habit of letting new planes enter when an existing plane is right by the edge so they crash before you can do anything. A warning icon does appear to give you time to move a plane out of the way, but it’s frustrating to lose a game in what seems like such an unfair manner. Finally I think the game could be improved by putting ticks on the planes paths so you see more easily when they well get to a certain point on the map. A small marker every five seconds of flying time would be very useful.

The game is a great pick-up-and-play title, and you won’t be able to play it just the once. With the game currently selling for a greatly reduced price it should be on every casual gamer’s iPhone.

Last.fm Chart Changes

2009-05-28T11:16:08+00:00

For several years I’ve written and maintained a GreaseMonkey script which adds chart change information to your music charts. The biggest problem with a greasemonkey script is that you don’t control the page you’re modifying. Last week, for the umpteenth time, Last.fm changed their page again and broke the script.

Fortunately, I’ve fixed the script and have taken the opportunity to improve the webservice that it uses. This means that the charts should be more cachable to improve performance for you and reducing bandwidth usage for me. I’ve also added support for weekly charts so they’ll now have chart change information, as they used to before Last.fm’s most recent redesign removed it.

Finally, because my host Linode.com recently increased the disk space on all their plans by a third, I’m able to increase the length of time all charts are stored for 30 to 120. Unfortunately as I had to delete all the chart change information you won’t see a change initially but gradually you’ll see your charts are available for longer and longer.

You can download the updated script here.

Enjoy!

Strict Development

2009-05-07T11:28:34+00:00

While working on my new open source project, CouchQL, I’m being very strict with my development process and following both issue driven development, and test driven development.

Issue driven development requires that every commit refers to an issue that has been logged in the bug tracking software. This means that every change must be described, accepted and then logged. This works better if your repository is connected to your bug tracking software such that any commit message with a issue number is automatically logged. In subversion this can be achieved with a post commit hook, such as this script for trac.

The connection between your commit messages and bug tracking software means that when changes are merged between branches new messages will be added to the issue, informing everyone what version of the software the issue has been fixed in. As well as just adding comments to issues it is also possible to mark bugs as fixed with commit messages such as “Fixes issue #43.” which should speed up your work flow. While Google Code does add hyperlinks between commit messages and issues, it doesn’t add automatically add comments, which is a pain.

Enforcing a development practice like this requires you to think about the changes you are making to your software, and focuses your mind one a particular goal. Bug tracking software will have the ability to assign priorities to changes as well to group them into milestones. This helps you to build up a feature list for each version of the software, and to know when you’ve achieved your goals and it’s time to release!

Test driven development is related in that before any changes to code are made a test must be written. This test must be designed to check the result of the change as closely and completely as possible. When the test passes (and all other tests still pass) the change that you were making is complete.

The benefits to this style of development are two fold. Firstly it should be easy to end up with test coverage of close to 100%. Secondly it forces you to think about the point of the change that you’re making. Combine this with the issue that you logged before you started, and you’ll really have a good idea of the scope and aim of your change before you ever touch the keyboard.

It can sometimes be hard to do this on every change you make, but the more often you do the better tested and more maintainable your code will be. Tools can really help you. If your version control system is linked to you bug tracking software then all you need to do is remember to log a bug and mention it in every commit message. A continuous integration testing tool such as buildbot makes keeping your tests complete very desirable as you’ll be notified of any breakages very quickly.n You can’t be made to follow development processes such as these, but if you understand the benefits, and want to use then they become second nature, and hopefully you’ll be a better programmer as a result.

CouchQL 0.1 released

2009-05-02T09:22:51+00:00

I’ve just uploaded the first release of CouchQL. It can be installed from PyPI by typing “easy_install couchql” or you can download a tarball from Google Code.

It’s a very early release, but please play with it, break it and email me your results!

CouchQL development progressing

2009-04-30T11:14:54+00:00

As I mentioned in a previous post I have been working of a library to ease the creation of map/reduce views in CouchDB.

The code is being hosted on google code and can be checked out and used now. The development is currently at a very early stage, but the fundamentals are sound.

Code such that given below will work. In this example it will return all the documents with a member ‘x’ whoes value is greater than one.

c = db.cursor()
c.execute("SELECT * FROM _ WHERE x > %s", (1, ))
for doc in c.fetchall():
    # process doc

The code is executed as a temporary view, but very high on my list is to use permanent views for much higher performance. This will be added before a first release, as will the ability to have multiple expressions anded together in the where clause.

Building Better Web Services With Django (Part 2)

2009-04-23T11:28:00+00:00

In the first part I talked about using the Content-Type and Accept HTTP headers to allow a single website to be use both by humans and programs.n In the previous part I gave a decorator which can be used to make working with JSON very easy. For our use though this isn’t great because a view decorated in this way only accepts JSON as the POST body and only returns JSON, regardless of the HTTP headers.

The decorator given below relies on a django snippet to decode the Accept header for us so don’t forget to added it to your middleware.

def content_type(func, common=None, json_in=None, json_out=None, form_in=None):
    def wrapper(req, *args, **kwargs):
        # run the common function, if we have one
        if common is not None:
            args, kwargs = common(req, *args, *kwargs), {}
            if isinstance(args, HttpResponse): return args
        content_type = req.META.get("content_type", "")
        if content_type == "application/json":
            args, kwargs = json_in(req, json.loads(req.raw_post_data), *args, *kwargs), {}
        elif content_type == "application/x-www-form-urlencoded":
            args, kwargs = json_in(req, req.POST, *args, *kwargs), {}
        else:
            return HttpResponse(status=415, "Unsupported Media Type")

        if isinstance(args, HttpResponse): return args

        for (media_type, q_value) in req.accepted_types:
            if media_type == "text/html":
                return func(req, args, kwargs)
            else:
                r = json_out(req, args, kwargs)
                if isinstance(r, HttpResponse):
                    return r
                else:
                    return HttpResponse(json.dumps(r), mimetype="application/json")
         return func(req, args, kwargs)
    return wrapper

So, how can we use this decorator? Let’s imagine we’re creating a blog and we have a view which displays a post on that blog. If they user posts it should create a new comment. Firstly we create a function, common, which gets the blog object and returns a 404 if it doesn’t exist. The return of this function is passed onto all other functions as their arguments.

def common(req, blog_id):
    try:
        return (get_post_by_id(int(blog_id)), )
    except ValueError:
        return HttpResponse(status=404)

Next we write two functions to handle the cases where the users POSTs a form encoded body, or some JSON. The return values of these functions are passed onto the chosen output function as the arguments.

def json_in(req, json, blog_post):
    # process json
    return (blog_post ,)

def form_in(req, form, blog_post):
    # process form
    return (blog_post, )

The JSON output function doesn’t need to return an HttpResponse object like a normal Django view because the output is automatically encoded as a string and wrapped in a response object.n

def json_out(req, blog_post):
    return blog_post.to_json()

Finally we come to the HTML output function. This function is also called if not mime type in Accept is suitable.

@content_type(common=common, json_in=json_in, json_out=json_out, form_in=form_in)
def blog_post(req, blog_post):
    return render_to_template("post.html", {"post": blog_post})

This decorator is really little more than a sketch. Many more content types could be supported, but hopefully it gives a good example of how you can write a very flexible webservice and still reduce code duplication as much as possible.

Introducing CouchQL

2009-04-16T11:49:40+00:00

CouchDB is a very exciting development in the world of databases and I’m greatly enjoying building a website which uses it. One problem is that most of the of views that I have created are extremely simple and could easily be represented using SQL. Although I wrote some code to help make life easier, creating a view such as that below is never going to be as simple as including SELECT * FROM table WHERE (status="open" OR status="accepted") AND latest AND key="xyz" directly in your code.

function (doc) {
    if((doc["status"] == "open" || doc["status"] == "accepted") && doc["latest"]) {
        emit(doc["key"], null);
    }
}

The SQL above and the Javascript view function are directly equivalent, which is why I’ve started working on an extension to the Python CouchDB library, which I’ve decided to call CouchQL.

The basic strategy is going to be this. The library adds a method to the Database object, cursor which returns an object which is compatible with the standard Python database API. When executing a CouchQL query a hash is taken of the textual query and a call is made to the view couchql_<i>hash</i>. If the view is not found then the query is turned into Javascript, added the server and the call repeated.

One of the common mistakes with CouchDB is to treat as if it were a traditional RDBMS. CouchQL has the danger of confusing people even more by allowing users to query CouchDB as if it is an RDBMS. CouchQL is not SQL, even if it does pretend to be SQL-like. I’ve not yet decided on how much processing should be done in the library to make the query language more SQL-like. The query SELECT * FROM table WHERE x > 5 OR x < 3 cannot be directly represented as call to a CouchDB view. It can be represented as two separate calls to the same view with the results merged. Is this a good idea? I’m not sure.

Development work has only just started on this library, but I’m actively working on and hope to be able to announce something useful to the CouchDB mailing list soon.

Building Better Web Services With Django (Part 1)

2009-04-08T11:21:03+00:00

Building a RESTful webservice is pretty straight-forward with Django, but in many cases you want to have both a human readable website and a machine readable api. A lot of websites solve this problem by using www.x.com as the human site, an api.x.com as the machine site. They also will typically have different structures to support the different usecases.n Unless your documentation is really excellent and the person writing the client to your service actually reads it building a client for the service is an error prone process. In an ideal world the developer would be able to browse the website and use the same urls in their client program. Fortunately HTTP has two headers which make it possible to do just that, Content-Type and Accept.

The Content-Type header describes the type of data that is included in the body of the HTTP request. Typically this will be values such as text/html, application/json or application/x-www-form-urlencoded. A content type is sent by the client when POSTing or PUTing data, and whenever the webserver includes some data in its response. The Accept header is sent by a client to specify what content types it can accept in the response. This header has a more complicated format that Content-Type because it can used to specify a number of different content types and to give a weighting to each.

When combined these two headers can be used to allow a normal user to browse the site and to allow a robot to make api calls on the same site, using the same urls. This makes it easier both for the creator of the programmer accessing your site and for you because you can easily share code between the site and your api.

I’m going to outline a decorator that will let write a webservice such as this, that will support HTML and JSON output, and JSON and form encoded data as inputs.n First we’ll create a decorator that parses any post data as JSON and passes it the view as the second parameter (after the request object). It will also JSON encode any return value that’s not an HTTPResponse object.

import simplejson as json

from django.http import HttpResponse

def json_view(func):
    def wrap(req, *args, **kwargs):
        try:
            j = json.loads(req.raw_post_data)
        except ValueError:
            j = None

        resp = func(req, j, *args, **kwargs)

        if isinstance(resp, HttpResponse):
            return resp

        return HttpResponse(json.dumps(resp), mimetype="application/json")

    return wrap

This decorator should be pretty easy follow, but here is an example to illustrate its use.n

@json_view
def view(req, json, arg1, arg2):
    obj = get_obj(arg1, arg2)
    if req.method == "POST" and json is not None:
        # process json here
        return {"status": "ok"}
    else:
        return {"status": "failed"}

This really cuts down on the code you need to write, but this view only handles JSON as its input and output. Next we need to parse the Accept headers and return an ordered list of content types so we can choose the preferred option. No need to reinvent the wheel, so we just pull some code from djangosnippets.org.

All the parts are in place now, and in my next post we’ll create a decorator which takes these parts ands puts them together.

Wolfenstein 3d iPhone Review

2009-04-02T11:27:33+00:00

ID Software have taken time out from letting you play Quake 3 in your browser to release a port of the granddaddy of all first person shooters - Wolfenstein 3d. The iPhone is really beginning to show itself as an excellent gaming platform, and despite its age Wolfenstein really looks at home on the phone.

The graphics are, quite frankly, rubbish. However, they’re exactly as they were when the game was released in 1992 which is exactly the point. The sound has faired much better and sounds great. The voices of the German soldiers still send shivers up your spine. It’s an extremely faithful port of the game, and the addition of an automatic save feature means it really works as a pick up and play game. It’s very easy to dip in and out of taking one level at a time.

The controls have naturally been revamped for the iPhone’s unique control system. You steer by moving your left finger over up/down/left/right arrows and fire by tapping your right finger on a button. This works really well and it is very easy to pick up and to start running around the Nazi prison that you find yourself in. The controls do rather feel like driving a fork lift truck. You regularly find your self reversing backwards around a corner, which is not how a person would move, but it’s easy to forgive and hard to see how else it could be done.

The game might seem a little expensive at £2.99 but you get all six episodes of the game which will last you many hours. It’s available in the App Store now, and for such an old game still shows its class amongst the other games available on the iPhone.

If you’re interesting in the process of making the port then John Carmack has written a detailed post describing it.

"Programming Collective Intelligence" Book Review

2009-03-24T13:31:57+00:00

Toby Segaran’s book is given the subtitle “Building Smart Web 2.0 Applications”. It’s clearly been assigned to the book by a marketing drone who felt that they needed to get the phrase “Web 2.0” in there somewhere. Unfortunately this left me feeling a bit disappointed when read this book because it’s not really what the book is about. A far better subtitle would have been “An Introduction to Machine Learning”, but I don’t work in marketing.

My expectation based on the title was that the book would focus mainly on how to generate recommendations based on a user interactions with a set of objects. I was expecting it to cover not only the mathematical basis for such algorithms but also the practical implications that sites like Last.fm and Amazon have to deal with processing such enormous data sets. While the book does cover recommendation systems they’ve given only a small slice of what is already quite a short book. There is no mention of how to process huge datasets either through map/reduce or some other scalable architecture.

The rest of the book is take up with classifying objects in a set, predicting the prices of a new objects given similar objects or filtering items such as email into spam and ham. These are staples of machine learning textbooks, but here are introduced in a very accessible and easy to read way. These chapters are interesting and informative but perhaps not that useful if you’re building a web 2.0 site.

This is a great addition to the O’Reilly stable of programming book, it’s just a shame they tried to shoehorn Web 2.0 in when that really gives the wrong impression of the book.

Programming Collective Intelligence Buy from Amazon (contains affiliate link)

iPhone 3.0 Thoughts

2009-03-18T13:34:28+00:00

Yesterday, as was splashed all over the web, Apple announced the iPhone OS 3.0. The key features in my view that were announced are cut and paste, push services and the possibility of turn by turn navigation.

I was a bit skeptical over being able to copy and paste. I imagined that it would only be useful if your were using the iPhone for business, which I don’t. Apple’s implementation seems to be highly impressive though, and I’m suitably impressed. Rather than just being limited to copying text between the inbuilt application it appears that you will be able to copy HTML, images and more between both the inbuilt applications and more. If you want to send more than one photo in an email just copy it from the photo app and paste it straight into an email. Sweet.

Recently my iPhone has been suffering from a strange problem where the battery goes from full to empty in just a few hours and feels warm to the touch. I suspect that the problem maybe due to the mail app so it is excellent news that the much delayed push services will be coming this summer. Rather than applications needing to poll for information developers can ‘push’ updates out to your phone automatically. This has the dual benefit of saving battery power and giving some applications a fake running in the background mode. The facebook app could keep you logged into facebook chat all the time and alert you whenever someone sends you a message. The same is true for email and other chat applications.

Apple are also opening up the iPhone’s mapping capabilities to give developers the possibility of creating sat-nav beating turn-by-turn navigation systems. There are some caveats, such as that developers will need to negotiate their own licenses with Google to use Google Maps or use Open Street Map to provide the mapping data. They will also need to come up with a way to cache maps on the phone so if you’re out of mobile range then you’re not lost as well. I expect that both problems will be quickly overcome and this could prove to be a killer application for the iPhone. I just a need a car…

Other applications include a landscape keyboard, greatly improve SMS support and MMS so you can text photos around. Unfortunately there is no date for release yet, but it will be released by this summer. It’s a free update for iPhone users but iPod touch uses will need to shell out a small amount of cash to get it.

Updating CouchDB Views In Django

2009-03-11T13:24:36+00:00

CouchDB views are a bit like stored procedures in a traditional database system. As with stored procedures it’s difficult to keep them in sync with your code, and to keep them in your version control system. In this article I’ll show you how you can use a django management command to update your views from files in your code base.

CouchDB uses a map/reduce system where each view is made of a filter program (the map) and an optional post processor that runs over the output of the map (the reduce). These pairs are grouped into design documents which are stored as a single unit in the couchdb database.

This command assumes that you store your map and reduce functions in the directory structure set out below.

project/
    app/
        couchviews/
            database1/
                design1/
                    mapreduce1/
                        map.js reduce.js
                    mapreduce2/
                        map.js
                design2/
                    mapreduce3/
                        map.js reduce.js
             database3/
                 design3/
                     mapreduce4/
                         map.js reduce.js

Inside your app directory create a folder called couchviews. Inside that create one for each of your CouchDB databases. Finally, create two layers of directories to represent the design documents and views stored within. Each javascript file should contain a single anonymous function.

For this management command to work your settings file needs to contain a variable for each database, containing the Python CouchDB database objects. In this example three variables need to be added to ettings.py - database1, database2 and database3.

Add the code below to the file project/app/mangement/commands/updatecouchviews.py and when you type manage.py updatecouchviews it’ll walk your directory structure and update all your design documents in one fell swoop. Easy!

import couchdb
import glob
import os

from django.core.management.base import NoArgsCommand

class Command(NoArgsCommand):
    help = "Update couchdb views"

    can_import_settings = True

    def handle_noargs(self, **options):
        import settings

        couchdir = os.path.realpath(os.path.split(__file__)[0] + "../../../couchviews")

        databases = glob.glob(couchdir+"/*")
        for d in databases:
            if not os.path.isdir(d):
                continue

            db = getattr(settings, d.split("/")[-1])

            for design in glob.glob(d + "/*"):
                design = design.split("/")[-1]
                try:
                    doc = db["_design/" + design]
                except couchdb.client.ResourceNotFound:
                    doc = {"_id": "_design/" + design}

                doc["views"] = {}
                for mapreduce in glob.glob(d+"/"+design+"/*"):
                    mapreduce = mapreduce.split("/")[-1]
                    mr = {}
                    mr["map"] = open(d+"/"+design+"/"+mapreduce+"/map.js").read()
                    try:
                        mr["reduce"] = reduce = open(d+"/"+design+"/"+mapreduce+"/reduce.js").read()
                    except IOError:
                        pass

                    doc["views"][mapreduce] = mr

                db["_design/" + design] = doc

Creating Django Management Commands

2009-03-06T13:32:51+00:00

Creating a website with Django is great fun, but eventually you’ll need to write a tool to clean up you data, expire old users or one of the myriad of other administration tasks that are involved with running a website.

You’ll be very used to using manage.py to create your database and to run your webserver. It make sense to use the same script for your own admin tools. Not only to get the benefit of sharing lots of you code but also Django will take care of parsing command line arguments and importing your settings for you.

Unfortunately the Django documentation is quite lacking on how to add your own command, but it’s really quite easy.

If your app is in project/app then create the directories project/app/management/commands. Create an empty file called __init__.py in both the management and commands directories. Once you’ve done that every other python file in the commands directory can be executed via manage.py. If you create the file project/app/management/commands/x.py then it can be run as manage.py x.

Each file which contains a command must define a class called Command which derives from django.core.management.base.BaseCommand. Rather than derive directly from BaseCommand you’ll most likely want to derive from NoArgsCommand, LabelCommand or AppCommand. First I’ll explain how to make a command when deriving from BaseCommand and then we’ll look at the helper classes.

from optparse import make_option
from django.core.management.base import BaseCommand, CommandError

class Command(BaseCommand):
    option_list = BaseCommand.option_list + (
        make_option('--long', '-l', dest='long',
            help='Help for the long options'),
    )
    help = 'Help text goes here'

    def handle(self, **options):
        print "This is a command"

The code above should be fairly self explanatory. The code for your command goes inside the handle function, while the options that your command accepts go in the options_list variable above. In this example the code accepts one option, long, which will be passed as a value in the options dictionary.

If you command doesn’t need any options then you should derive from NoArgsCommand and override handle_noargs rather than handle.

The other two types of commands are AppCommand and LabelCommand. These both take any number of arguments on the commandline, but AppCommand checks that they are app names while LabelCommand passes them on unchanged. You should override handle_app and handle_label rather than handle, and these functions will be called once for each commandline argument with the argument as the first parameter to the function.

For more inspiration take a look at the default management commands

Next time I’ll show how you can write a real command which will help you keep your views updated if you use couchdb as a datastore.

Sony DPF-D70 Digital Photo Frame Review

2009-02-26T13:31:40+00:00

Technology is finally catching up with the principle of a digital photo frame and this Sony DPF-D70 is really good example. The screen is fantastic, really clear and bright which shows off your photos well. Getting your photos on is a snap. Simply plug in your camera’s card and click “Add to Album”. It features 256MB which can hold a lot photos at the frame’s one megapixel resolution.

The frame comes with a range of options. You can change how long the photos stay on the screen for, how they appear. You can display single photos, three photos at a time or clock or calendar view. It also features a handy timer function so you don’t need to worry about it using power all night.

On the downside the frame must be plugged into the mains as there is no battery option. Also Sony have skimped on the packing as despite having a mini USB connector no cable is included in the pack. Come on Sony, can’t you stretch to a simple cable?

Disappointingly the firmware seems to have few bugs. The menu system gives you the option to rotate a photo but when you turn the frame off the photo returns to its original orientation. This leaves that feature feeling a bit pointless.

If you want a largish frame with a great screen then this Sony should definitely be on your list of possibilities.

Darkslide Flickr iPhone App Reviewed

2009-02-19T13:12:35+00:00

Darkslide is an excellent iPhone app which makes it easy to use Flickr on the move. The iPhone is a perfect partner for Flickr. Constant internet access, GPS and serviceable camera mean that you can snap away and have your photos on the internet and geotagged in seconds.

Flickr does have a mobile site and provides it user with an email address where you can email photos. On an out of the box iPhone it’s possible to get a decent Flickr experience. Emailing your photos has several disadvantages: the quality is reduced, and the photo is stripped of geotag information.

Darkslide lets you easily take and upload photos at full resolution. You can also tag, comment and browse through both your photos and others. You can also access the Flickr’s interesting feed if you’re in need of inspiration. It’s available both with and without ads, but apart from that the two versions are identical.

It’s simple, slick and just works. If you’re a Flickr user with a iPhone then it’s a must have.

Download Darkslide free.

Download Darkslide Premium ($3.99).

Google Sync For iPhone

2009-02-12T13:32:35+00:00

Google released an excellent new tool for automatically syncing your iPhone’s contacts and calendars and GMail and Google Calendar. Although, as is usually the case with Google, the service is marketed as being in Beta it seems to be working very well. Add or edit a contact either on in GMail or on your iPhone and within seconds it’s been synced across. Even contact photos are synchronized. The same is true of calendar entries.

Rather than implement a new syncing application for the iPhone Google have just linked your contacts and calendars to a Microsoft Exchange server. Apple implemented push services from Exchange services in the 2.2 firmware to help market the phone as business capable. Now though, anyone who uses GMail can benefit from this technology.

Push services are useful because rather than your phone checking at a predetermined interval, probably to find that there has been no change, changes are ‘pushed’ to you. This means that not only are changes that do happen reflected much more quickly on your phone, and you don’t waste battery life or network traffic checking unnecessarily. Unfortunately it doesn’t support email syncing, but surely that’s only a matter of time?

Visit http://m.google.com/sync to get your iPhone set up.

Flickr Five Years Old Today!

2009-02-11T13:20:39+00:00

Today is Flickr’s fifth birthday. I’ve had a Flickr account for several years, and I use the site all the time. For sharing photos, finding photos to use on a website, or just to browse around it’s really unmatched.

Happy Birthday Flickr!

Rude Code

2009-02-05T13:28:08+00:00

In this second part of my “Etiquette of Programming” series I’m going to talk about making sure your code fits in with the style of existing code, while helping to bring it up to best practice standards.

When you’re working on an existing codebase, either fixing bugs or enhancing features, you’ll be going into the code and adding more code. When someone has to come back to this section and improve the feature or fix more bugs (not that you would introduce any, would you?) they need to read both the original code and your code. If your code sticks out, or causes the reader to say “Whoa, what’s going on here?” then your code is rude, and no-one likes rude people do they?

Just like writing prose, everyone has a different style of writing coding. People prefer different programming paradigms, design patterns, variable naming schemes and have different aesthetic preferences for spacing their code out with white space. There are more differences, but these four are the main aspects that make up your style of code. I’m going to talk about all four aspects in turn, from rudest to nicest.

Most people code in an object orientated style, but most languages allow you to pick and choose between procedural and functional styles as you wish. Each style has its own benefits and drawbacks, and each should be used in the right place. What you should avoid is switching styles in the same section of code. The mental switch required for someone who is reading code written in multiple styles is too great for it to be an easy or enjoyable experience.

If you choose to write your code in an object orientated style the you could chose to use any number of design patterns. From factories and singletons to full blown class hierarchies design patterns can be incredibly complicated. They all solve essentially the same problem though. They aim to help you structure your code better to give you a more reliable and more maintainable program. An existing code base will usually have evidence of design patterns being in use. Whatever you think of the particular patterns in use, if you want you code to be maintainable then it’s important to use the same patterns.

The final three points are not as important as the first two, but combined they can really cause great difficulty for those reading and extending your code. Most of the time you’ll be using APIs that you don’t know too well. You’ll probably now that there is a function or variable that does what you want, but you’ll need to guess at its name, or look it up. Consistent naming schemes greatly aid in remembering all the nooks and crannies of an API and reduce the mental effort needed to code using it. Typically you’ll be using several different libraries which will have different naming schemes, but if you can be as consistent as possible in the code that you control you’ll have more room in your head for more interesting things.

Lastly we have whitespace. In most languages whitespace is unimportant, and even in languages such as Python where it is, they are pretty forgiving about how you format your code. Four or two space indentation. Braces on the same or next line. Spaces around operators. The variations in style are enormous, but when you’re making a bug fix resist the temptation to change the style to fit your view because it will obscure the change that you’re actually making. If you need to reformat code it should be done in a single, dedicated revision which is clearly marked as only affecting white space.

For code to be maintainable and extendable it ideally looks as is it was written by just one person. None of these points are meant as hard and fast rules. Rather they are something to bear in mind to try and help you rein your natural desire to make a mark on the code. Best practice dogma changes over time and code should evolve with it. However, it should evolve in a dedicated refactoring step not piece by piece as new features are added.

Next time I’ll talk about the etiquette of source control and how it should be used to make your colleagues lives easier.

Using Django Forms For GET Urls

2009-02-03T13:25:04+00:00

A regular occurance in writing webapps is the user will submit a form, and on the results page you’ll want to include a link which the user can click to resubmit the form. This lets users bookmark the page or you can add an extra parameter such as ‘format’ so they can download the results.

While Django forms contain several functions for converting the form to HTML, it doesn’t contain one to convert a bound form to url arguments.

Fortunately Python’s standard library module urllib contains a module which converts a dictionary to a properly formatted url argument string. We simply add a function, as_url_args, which passes the form’s cleaned data to this function and we get back a nice string we can add to a link.

import urllib
from django import forms

class MyForm(forms.Form):
    name = forms.CharField

    def as_url_args(self):
        return urllib.urlencode(self.cleaned_data)

To use this in a template, where form is the form object, we can add it to a link.

<a href="/form?&format=csv">Download As CSV</a>

Dynamic Initial Values in Django Forms

2009-01-28T13:33:14+00:00

I recently had cause to create a form with two date time fields which had the default values of midnight seven days ago, and midnight this morning. Initially I thought this would be easy and created the following form.

from datetime import datetime, date, timedelta

class MyForm(forms.Form):
    date_from = forms.DateTimeField(label="From",
                               initial=(date.today() - timedelta(days=7)))
    date_to = forms.DateTimeField(label="To", initial=date.today())

This works fine except that when a process has been running across more than one day the initial values are no longer correct as they refer to the day the process started. Fortunately it appears that there is an undocumented feature where the initial value can be a function rather than an absolute value. This function is called each time the unbound form is displayed, so they are always correct.

Wrapping the code to create the value in a lambda works great here, as does passing a reference to a function.

from datetime import datetime, date, timedelta

class MyForm(forms.Form):
    date_from = forms.DateTimeField(label="From",
                 initial=lambda: (date.today() - timedelta(days=7)))
    date_to = forms.DateTimeField(label="To", initial=date.today)

Placing Django Models In Separate Files

2009-01-20T13:20:15+00:00

Chris Petrilli has made a very useful post on placing Django models into separate files.

The first thing I do when starting a Django project is to delete the standard views.py file and replace it with a directory. It won’t take you long before you’ve written enough views that a single file becomes huge. The same is true of models.py. If you have ten or more models then the file can quickly become a thousand line behemoth. I’d tried to split the file into a directory before, but it never worked, and the error messages were never helpful.

The solution is simple, you add a Meta class to your model with an app_label parameter.

from django.db import models

class Test(models.Model):
    class Meta:
        app_label = 'myapp'

So, thanks Chris!

The Etiquette Of Programming

2009-01-15T12:57:46+00:00

You might be a smart developer. You might be someone who gets things done. Unless you’re a lone programmer working in your bedroom though, that’s not enough. Collaborating and cooperating with your teammates is vital to keeping your project moving forwards and to ensure it’s extendable and maintainable.

In this series I’m going to talk about how small changes to your coding style, small changes to your communication and small changes to your working practices can make a huge difference to those you’re working with. A lot of these tips are common sense, some of them you’ll already know, but hopefully some of them will make you think and might make your life easier.

If you’re fresh out of university and joining a team of programmers with 30 years experience then you will probably get there and think “WTF!” about some aspect of their working practices. Most of what you read about programming is of the “here’s a shiny new toy” variety. If you immediately started using everything you read about not only would you probably go insane but you’d also annoy your coworkers immensely.

In a project with a reasonably sized team, especially one with people of different abilities, consistency is key. If you’re starting a project in a new company which has never written software before then you’ll have the opportunity to do things right. If you’re rewriting an old project from scratch then not only can you improve the code but also you can improve the working practices too. The first of those possibilities is unlikely, while the second is rare. What is much more likely is that you’re working either to maintain or to incrementally improve an existing code base.

Improving code and practices is an incremental process. Try to change too much too quickly and your coworkers will find it hard to keep up with the changes given the mixed state of the code base. How to stage the transition from old and broken working style to best practices is something I’ll discuss later.

Vim or Emacs, Windows or Linux, Mac or PC. Computers are full of ideological wars that will probably never reach a cease fire. This is a good thing. Arguments spur people on to ensure that their chose side is unquestionably better than their foe, and we all win as a result. When you’re working in a team though, arguments of this nature are trivial and can undermine the success of your team. In a future post I’ll talk about how to control these arguments so they remain productive.

In my next post, I’ll talk about rude code, how to deal with it and how to avoid writing it.

Working in Django, finally!

2009-01-14T14:08:22+00:00

For the past day and a half I have been building a website in Django rather than the C++ that is the bulk of my day job.

It’s so easy, it makes me sad that I have to go back to C++ this afternoon.

I’ve also discovered that aggregation, the main feature I want from django, will be committed to trunk tomorrow. Hurrah!

iPhone podcast gripes

2008-11-24T13:31:13+00:00

I’ve been using my iPhone for two weeks now, and overall it’s great. Easily the best phone I’ve ever used. It really has changed the way I live my life because there is so much I can now do from anywhere that before I’d need to be at my computer for.

Despite this though, there are a few things that Apple should fix.

There’s no character count when typing a text message. This might be a small issue and given I get 500 free texts a month (which is way more than I use) it doesn’t matter that I might sometimes send people two messages. However, it would be nice to be able tailor my messages to the limit of 160 characters.
The new over-the-air download of podcasts is completely broken.

Rather than listen to my music on a loop while I travel I listen to a number of podcasts, and Steve Lamacq’s 6music show. MythTV on my PC records his show every afternoon. A script then converts the recording into a 300MB MP3 file and updates the xml document that describes it as a podcast. This is served by Apache on my local machine.

Specifying http://192.168.0.8/podcasts/feed.rss works fine for iTunes on my PC. When I sync my iPhone new episodes of Lamacq’s show get copied automatically across.

It was well known that an update last week would add over the air downloads for podcasts. This would be brilliant I thought, because then I wouldn’t need to turn my girlfriends laptop on unless I want to back my phone up.

Disappointingly there are two big flaws with the implementation of podcasts on the phone when compared to podcasts on iTunes. Firstly there is no automatic checking for new episodes - you must press a button for each feed to see if it has been updated. Secondly only podcasts that are delivered through the iTunes store can be downloaded straight to your phone. When attempting to sync my custom podcast on my phone no attempt is made to download the feed and I get sent to the store’s podcast homepage.

Is there any need for this limitation? I can’t think of a good reason for it :-(

Google is useless to see if anyone else has come across this issue or resolved it because it’s full of speculation and screenshots from before the update was released.

Compiling CSS

2008-10-21T12:30:35+00:00

A new compiler called HSS has been released. This tool makes it easier to write concise, maintainable and valid CSS. The two key features are the ability to define variables which are then replaced throughout the file, and to create nested blocks.

Typically you’ll need to write code like this:

.faq {
    color: red;
}
.faq h1 {
    background-color: black;
}
.faq a {
    text-decoration: none;
}

With HSS you can write…

.faq {
    color: red;
    h1 { background-color: black; }
    a { text-decoration: none; }
}

Since you’ll want to automatically run a code minimiser over your CSS code anyway, running HSS as well shouldn’t be a problem.

Unfortunately the tool is written in the little known language Neko (which is a language created by the author of HSS) and is not hosted on any of the of many open source project hosting sites which would help ensure the tool lives on even if the author moves on. Indeed, it’s not even clear from the website what license the tool is released on. These are problems that might need to be solved before it’s worth basing a project around HSS.

New Flickr Home Page

2008-09-25T12:13:18+00:00

Flickr are in the process of testing a new style home page for logged in users. The process of switching is pretty fancy, once you click ok the old page fades out and is replaced by the flickr throbber which floats down the screen like snow flakes.

The new page is not really that different to the old page, but it does have slightly more information on it. I’m not quite sure what the point of the redesign was - it was nowhere near as dramatic as Facebook or Last.fm’s recent revamps.

Flickr’s website has changed little over the past year so it’s nice to see that the website won’t stagnate, and that new features are being added.

CouchDB Performance

2008-09-19T12:17:31+00:00

I’ve been toying with CouchDB for a short while, and I’m definitely impressed by what I’ve seen. Once I’d upgraded to Erlang R12B and trunk CouchDB any bugs I was seeing disappearing and importing all 1 million documents was straightforward.

With 1 million documents the map/reduce takes a long time, as you would expect. What would be nice is if the maps could be spread across different nodes to speed things up dramatically. Once the map has been calculated and cached, retrieving it is relatively fast. Parsing it in Python does seem to be quite slow, taking a few seconds for a few tens of thousands of results. This is far too slow for a webpage response.

Is there any way to speed up CouchDB? Well aggressive use of memcache will probably help, but too me it seems that CouchDB is not suited to large datasets. I do hope I’m wrong though, and I’m going to investigate further because I really want to find a use for CouchDB in my work.

Andrew Wilkinson

LDX3 London 2025 (Day 2)

From dashboard soup to observability lasagna: Building better layers

Growing pains: Scaling and re-architecting systems under fire

Being secure by design: Engineer-led security

Balancing direction and empowerment

Neurodiversity: From struggles to solutions, tips for leading a team

What’s my job again? Developing self-management

So you want to hire engineering force multipliers?

Getting excited about maintaining legacy systems

A safe place to land: Practical (and hyper-scale tested) advice for technical decision-making while evolving systems in place

Mastering Managing, Mentoring, Coaching (Workshop)

LDX3 London 2025 (Day 1)

Escaping The Rewrite Trap

Rethinking Collaboration

How Monzo Tolerates Full Cloud Outages

Monolith-ifying Perfectly Good Microservices

Metrics, KPIs and Developer Experience: Rethinking Measurement For High Performing Teams

Beyond “Try Harder”: Effective Strategies To Tackle Bugs

Human-first Leadership, AI-Powered

Fast, Informed, Impactful: How To Master Decision-Making

Tracking Repository Health at Scale

Code Your Future

How To Delete Everything: The Clean-Slate Approach to Technical Strategy

Rebuilding at Scale: A CTO’s Journey in a High-Growth Fintech

Levelling Up: Transitioning Successfully into a Manager of Managers Role

Frictionless Movement: How Internal Mobility Transforms Engineering Culture

Lost and Alone over the Pacific

Water Monitoring

LeadDev London 2024 (Day 2)

Build great engineering teams and have a positive social impact

Revolutionizing engineer growth: The tech-powered blueprint for careers clarity at ASOS

Making smart investments: A framework for maximizing your ROI in technical decisions

Iterate to greatness: Building high performance, AI-native engineering teams

Beyond the hype: Practical steps to establishing and scaling your data & ML team

The Post Office Scandal: What we can learn from its process and human failures

The boss’s shoes don’t fit: And other surprises of leadership

Uncertainty of change

Managing the marathon: Leading teams through lengthy migration projects

The software bug all stars - and what we can learn from them

How to set goals with people who don’t want to set goals

Managing Engineering Teams in the Era of AI

If you’re not part of the solution, you’re part of the problem

Mentorship + sponsorship

The engineering leader’s playbook: A hands-on guide to DORA

LeadDev London 2024 (Day 1)

Embracing engineering’s place at the forefront of business

How do you deliver a feature on the biggest stage in the world?

Engineering leadership in 2024 and beyond: Skating where the puck is going when the ice is melting

Tech debt as innovation, reframing this forever problem as an opportunity

From overload to oasis: Creating more time for the cool stuff

Managing across timezones; A retrospective

Building for the new developer

Demystifying neurodiversity in tech with nostalgic video games

How to use technology radars to make transparent tech decisions

Delivery metrics - the good, the bad and the utterly ridiculous

Measure for change

Data science demystified

You are here: The story of the Barbican

What Do You Want From A Team Meeting? A Decision Or A Team?

Balancing Standardisation With Innovation

Pimoroni's Interstate 75W

Sustainable Software Development

Small Celebrations, Big Motivation

Leadership Stand-Ups

LeadDev London 2023 (Day 2)

Building bridges: The art of crafting seamless partnerships between engineering, product, and design

Riding the rollercoaster of emotions

Platform engineering is all about product

Cultural post mortems: an approach to learning and recovering when your people systems fail

Strategies for succeeding as a underrepresented engineering leader

Feature flags unleashed

Build a data-driven on-call workflow for your team with atomic habits

Keeping your team health after a layoff

Parents who code: How to welcome your developers back after parental leave

Creating inclusive career ladders

Building for the underserved, solving for all

Driving positive change through performance improvement plans

Exit plans and how to talk about them

Orchestrating thousands of bots from the cloud