When to use a custom search solution

This post also appears on the Headscape site.

Website search can often be a thorny subject. Expectations of search capabilities have been set very high by search engines. The default search offered by content management systems (CMSs) is often a bit basic by comparison. For example, the results are often not the most relevant to the search term used because the CMS is only offering a simple “does this page contain this word” method.

Our discovery work for the Competition Appeal Tribunal site revealed a specialised set of search requirements. When working with a CMS, I’m a firm believer that you go with the flow of its strong and weak points, and try not to fight it. With that philosophy in mind, my instinct was not to struggle to make native CMS search fit the requirements but rather to develop a custom solution for this site.

We opted for Elasticsearch, which is described as a “RESTful search and analytics engine”, combined with the very well maintained and documented Elasticsearch PHP library.

As soon as you put some data into Elasticsearch it does a good job of returning relevant results. But where it really excels is when you begin to fine tune it to your needs.

As an example, there is a field on the site that we needed to be able to perform full text searches and sorts. This can be accomplished by duplicating the field into two different “types” allowing one to have all of the full text search capabilities whilst the other is used for sorting.

On another project for US law firm Buckley Sandler (that is built on Drupal), the different content types and fields did not all have the same relevance in search results. By applying a “boost” to specific content types and fields we were able to deliver the most relevant results to users.  Whilst Drupal’s native search API does allow a boost to be applied to certain fields, using Elasticsearch we were able to boost specific content types, individual pages and taxonomy terms.

This level of customisation may not be required on your project, but if it is I highly recommend trying Elasticsearch as I’m confident it can meet your needs.

Chatbots, always available customer support or know nothings?

This post also appears on the Headscape site.

After reading Build Chatbots with PHP by Christoph Rumpel I started making some simple chatbots and quite quickly moved on to more complex interactions. It didn’t take long to get something basic up and running, which in turn, got me really excited about the possibilities.

But – there’s always a ‘but’ isn’t there – it also didn’t take long for the bots to fail to have answers, especially when I showed them to colleagues who didn’t know the precise phrases the bots were listening for.

To try and get over this I used Dialogflow for the backend as it has natural language parsing capabilities. It also has a very nice UI that will allow even non-technical users to build up conversations. After a short tutorial I found Dialogflow easy to use and very powerful for the little effort needed to start using it.


Highlighted by others asking my bot questions it did not understand, it is essential that you do in-depth user research before you launch your bot. The last thing you want is to spend a lot of time and money only for people to never use the chatbot as they had a bad first experience.

Just like any web project, speaking to the people who will be interacting with your site / bot will show you interactions you can never have imagined by yourself.


It’s highly likely that you run some analytics or monitoring code on your website to ensure users are moving through the site as expected. You should also be monitoring the chatbot’s interactions to see where users take conversations and if any of these lead to dead ends.

This could be in the form of capturing the sessions with a tool such as Full Story, or logging the conversations to a database that you can then analyse at a later date.


Machine learning is great but machines also need help as well. Whilst Dialogflow has a machine learning aspect to it there will more than likely be cases where a user asks a question that it cannot learn from. By spotting new forms of interaction early you can return to the research phase to see how others would phrase the same question and then feed these alternatives into the backend.

You may also find people are looking for specific areas of your site and your overall information architecture could do with some readjustment. Or better still, maybe people are coming to you looking for a service or product you do not yet offer but could do, leading to new lines of business.

Chatbots are an exciting development and give the potential to have a form of customer support that’s always on hand. But they are not something that can be set up and never looked at again, otherwise their lack of knowledge can leave users feeling frustrated and abandoning your site.

Blogging, Completed It Mate

Oddly enough I’d had this post in mind over the summer and then some gentle encouragement from Paul Boag on an episode of The Boagworld Podcast convinced me to finish writing it.

Towards the end of last year I had several ideas for blog posts rattling round my head. I made a conscious decision I should actually finish the blog posts instead of leaving them as “Draft” and thinking “I’ll finish that soon”.

Off To A Flyer

When you have lots of ideas the posts just flow, you have momentum and enthusiasm and you’re a blogger master.

I even started setting myself reminders and began blogging quite regularly. Then it stopped. Like so many people stop and I’m not sure why. I missed a week, promised I’d finish the post next week and then all of sudden I was out of the habit.

Why Blog

I had a think about what made me start blogging in the first place

Had some ideas
Self documentation
Sure up my own thoughts and processes
Help others

It was good to finally finish off some of the blog posts I had stored as drafts and that made me look at other aspects of my job and think “I could post about that”.

I’m sure most developers document parts of their jobs that they don’t do so frequently. My own website seems a good place to keep those documents.

A bit like rubber ducking the act of writing out a process really helps me to straighten out my thoughts and ensure I’ve covered all aspects of a job.

It’s been really rewarding to see links coming in from drupal.org and seeing some search terms people have used and ended up on my blog such as Testing SSL certificates without a domain name.

When to Blog

Maybe I aimed a little too high with once a week at the start and things started to feel forced.

My aim going forward is to commit to once a month but if I have a series of posts I’ll possibly look at releasing those a little closer together.

Ultimately I want it to be a thing I enjoy rather than forcing myself to do it.

Thanks To Anyone Who Blogs

I’ve learnt so much from blog posts over the years. People just sharing a little bit of knowledge they’ve learnt to people consciously documenting the process they’ve been through.

It’s also given me a huge appreciation of people like Paul Boag who have consistently produced lots of high quality content over a long period of time.

Maybe, just maybe, this blog post will make you write a blog post, and you never know maybe I’ll find that blog post and it will help me. Even if that never happens there is a good chance it will help someone, even if it’s just your getting your thoughts in order.

Today I am Not Working From Home

I’m a full time remote worker and have a room at home set up as an office I can shut myself away in. As I’ve talked about before I love working from home but this week for the first time I decided to try remote working from a public space. I choose a local Wetherspoons as it seemed to tick all the boxes and was a short walk from home.

Setting Off

The night before I actually made a list to ensure I’d have everything I needed. Small things like earphones that I don’t normally use should come in very handy in a public space. One last mental check before leaving home and I was off. Luckily it was a dry morning and having a walk first thing was very nice.

Essentials, Free WiFi and Power

A lot of places have free WiFi these days so I knew that wouldn’t be a problem but it was only on my way that I realised I would need a power supply to last the whole day, rookie error. I opted for the honest approach and just asked when I arrived. Handily there was a nice table tucked away at the back with a power socket.

What do I do When I Need the toilet?

Everything was going well and then I hit my first “What do I do now?” moment. When I need to pop to the toilet what do I do with all my stuff. By this time there was a table of retired gentlemen sat on the table beside me. I had protectors of my belongings.

Not as Many Screen Breaks

When I’m at home I find myself getting up to make a drink a bit more often and normally have few paces around the kitchen while I wait for the kettle to boil. Being somewhere where food and drink are brought to you I found myself sitting in the same place for a lot longer than I usually would.

Planning Calls

As I mentioned in my “Make working from home work for you and your team” post I like to make myself available for quick chats with colleagues. Today I had no client calls planned and didn’t end up needing to chat with anyone else. I had my earphones and I suppose I would just have had to try and talk at a polite volume had I needed to.

How Long Can I Stay

Then came the second “What do I do now?” moment. When I arrived I ordered food and drink, but now I feel like I’ve outstayed my welcome, and that was nothing to do with the staff just my internal thoughts. So I ordered another drink and a small bite to eat.

Overall Productivity

When lunchtime came I made sure to leave and have some time away from the “desk” just as I do at home. I got plenty done away from home and don’t feel it impacted my productivity at all. It was nice to be somewhere different and it adds to my enjoyment that I’m lucky enough to have a job where I can do such a thing.

Would I do it Again?

Yes, but certainly not everyday. Ignoring the cost of buying food and drink so I feel like I can stay, I do like being in the home office. Maybe I shall save it as a once a month treat.

Fractal – A Backend Developers Perspective

One of the great thing about attending conferences is attending a talk and learning about something you didn’t even realise you wanted to know about.

During DrupalCon Vienna I saw a talk by Anton Staroverov and Tassilo Groeper from Wondrous titled Decouple your Twig from PHP and make Frontenders happy! In the talk they showed a tool called Fractal which has been developed by Clearleft. After seeing the presentation I felt this would be a very useful tool so set about looking into it further.

Easy to Use

That evening I started to try out Fractal and within 10 minutes I had a build up and running and had added a component to it.

Since then at Headscape we have switched the default templating language to Twig to tie it into our Drupal 8 development. We’ve also used it’s static output to produce pattern libraries for clients.

Reference Point for All

Not only is Fractal a useful tool for frontend developers as mentioned above we also use it to show clients both individual components and components working together.

As a predominantly backend developer I also find it useful when I need to add a button for example. I can easily find all the button options in the nice frontend UI and see what classes I need to add.

Helps Building Templates and Reusable Components

Having one place where we can list all of the components and then using this as the actual templates for the production build is very useful for me.

During development, wether that is early stages or updates in the future, I find having a reference point of components speeds up the process.

For example if we have a new content type I can look at the various view modes we already have to see how I should be outputting the content.

With how quick it is to get something up and running with Fractal I’d encourage you to try it out. It might take a little bit of settings adjustment and maybe even splitting you’re templates and CSS out into more componentised methods but I think the benefits you’ll gain will be worth it.

Drupal 8 Workflow Notifications with Rules Part 3

In previous posts I’ve covered enabling and editing a Workflow and then sending an email when a new content moderation state has been saved.

In this post we will complete the loop by sending emails when a content moderation state has been updated.

In this example an editor will publish the node and the rule will email the node creator to tell them it has been published.

Head to the Rules admin page /admin/config/workflow/rules and click Add reaction rule

This time we want to choose After updating content moderation state from the React on event select.

Adding a rules event for updating content moderation state

This time round we will need two conditions.

  1. Check the unchanged moderation state
  2. Check the new moderation state

Click the Add condition button and then choose Data comparison from the select.

For the data to compare value you can either use the data selection or enter


Then for the data value enter review

This is the moderation state the node was in before it was saved.

Now click save.

Moderation state unchanged data comparison condition

Add another data comparison condition then this time enter the following

Data to compare value


Data value


New moderation state data comparison condition

Next we need to fetch the entity so we can use it’s values in our email.

Back on the edit page for this rule click Add action and choose Fetch entity by id from the select.

For Entity type enter node and for the Identifier value enter


Then click save.

Now we can send an email to the node creator.
Click Add action and choose Send email.

In the Send to field use direct input and enter


This is a token that gets replaced with the users email address.

For subject we can just enter something appropriate such as Published Notification

For the message we will use the direct input mode an a token that provides the entity title

Your content {{entity_fetched.title.value}} has been published.

Email action

Save that and then save the rule.

As is often the case with Drupal it’s worth clearing your cache before testing.

Rules is still in it’s infancy for Drupal 8 but I feel it provides enough functionality and is well enough supported that it can be included in production site.

Why Your Back End Developers Should Be In The Design Kick Off

Design kickoffs are great, it’s the start of the project, expectations are high, ideas are thrown around liberally and everyone is excited about what’s to come. So why should you bring along the person responsible fo database tables and code no one will ever see?

Get to know the business

Invitations to tender and design briefs are very useful documents to get an initial idea of what a business does and what they offer. You may even be able to gather some useful information from their existing website, or you may not and that maybe the reason you’ve been brought in.

Having attended many kickoff meetings now it has become obvious to me that you only really get to know the goals of a business when you speak to those that work there.

As an example the company streamline may be “We provide widget X to 90% of the industry” but when you speak to the web team the real problem is embedding YouTube videos means copy and pasting embed code into a text box. You aren’t hired to try and sell more widgets, you are there to help the web team be able to sell more widgets.

Overall knowledge of project

“We want to modernise the look and feel of our current site and make it responsive” this may be the driving force behind project but more than likely there are many other goals that the client has in mind that can easily be accomplished within scope and budget.

I can guarantee that in the meeting someone will jokingly complain about a current process that they have to go through that can easily be automated in the new project. When offering a solution to the client it is always met with a smile and a comment such as “that would save me so much time”.

Having a broader idea of the whole project can also help you set up for potential future work. Somethings may be identified that can’t fit in the current timescale but you can build in the foundations that will allow future updates to be made.

Possible technical hurdles

Whilst I love the excitement a new project brings it can be the case that enthusiasm takes over a little and it can be an excellent time to spot any potential technical hurdles that are flagged up. I have been on plenty of projects where it’s casually mentioned that content lives in this system or the data needs to be sent to another place for a contact database.

Some companies may have certain limitations on the technologies they wish to use. They may only want to use their own servers or a certain application stack to allow for easy integration with other services they use.

Forming a relationship

As I work for Headscape, who are an agency, rather than an in house team forming a relationship with clients early on is highly valuable for a good working relationship.

Once the kickoff is done I rarely see the client again face to face. We do however normally have at least a weekly meeting on Skype or GoTo Meeting. Having met in person I always find it helps these calls as you can at least picture the people and their mannerisms. In every project I’ve done the calls always have a friendlier tone to them after having met.

Gets me out for the day

Us backend developers may be stereotyped as shy people who want to sit in a room with no social interaction but if you are in two minds as to wether you should take your backend developer along to the design kickoff meeting I hope the above will convince you that it will really help the project if you do.

Customer Service as a Service

In my younger days I worked in a surf shop where the owner was very hot on customer service. He liked us to greet every customer when they entered and left. To start with this seemed quite unnatural and the odd customer even seemed confused. In general though it did open up communication and people seemed genuinely happy with the interaction even when you said thanks to them for leaving empty handed.

Now I’m a web developer and spend the majority of my days sat in a room by myself but customer service is just as important. With you’re building an agency building a site or working on an app you will have customers and they deserve to be treated well.

These days there are many developers and agencies out there that all do fantastic work. You can no longer rely on being a scarce resource. Providing great customer service goes a long way to making the project enjoyable and successful as well as paving the way for repeat business and recommendations.

Be polite

It may seem like basics you teach small children but remembering to be polite in the digital world so not be taken for granted. A simple “thank you” when the client helps by sending you the updated text or images you requested sets up a good working relationship.

It’s their site, they use it, try to accommodate them

You may not always agree with your clients requests and often this comes down to opinion. At he end of the day they are the ones who are paying for a website and will have to live with it everyday.

As an example there maybe a content management task with two options.

  1. Simple technologically but harder for the CMS user
  2. Harder technologically but makes the CMS user’s job easier

Always try to go for option 2. I often find it’s the small things that make people’s lives easier that are most appreciated.

They may not understand as well as you, try different explanations

If only clients could see what I see in my head. Everyone learns and understands differently and concepts that make perfect sense to me about a database can be baffling for others. There are many times you will need to take a different approach to explanations to help clients understand your concept.

For me nothing is better than quickly mocking something up so we are all looking at the same thing rather than relying on a fuzzy dewscription that can easily be interpreted wrongly.

Check they understand, ask if they are happy

Sometimes people end up agreeing when they aren’t 100% sure of the outcome. There is always lots to be thinking about in web projects. Taking a few seconds at the end of a discussion to ensure everyone understands the next steps and they are happy with them can save doing unnecessary work when the client reveals this wasn’t what they really wanted.

Give options

I find there is rarely a single option for solutions. Many times it’s a case of there being at least two ways we could achieve a task, both of which come with pros and cons.

Using the above advice provide the client with the options and a clear explanation of the pros and cons and let them decide their preferred implementation.

It not just for customers

Looking back at my time in the surf shop it didn’t take long before I was chatting to everyone I came into contact with and I instantly recognised how this simple act made all interactions easier and more enjoyable. All of the above points can be applied to your colleagues as well.

For example if a designer comes to you with an idea for an interactive menu there maybe two ways it could be implemented on the website. Discuss the merits of each idea and which will benefit the design the best and everyone will be happy. The designer gets to choose the best visual and you don’t get a technically impossible PSD.

Back End Developers – Goalkeepers of the Web

When I was younger I wanted to join a football team. I had visions of scoring the winning goal and being the hero. Unfortunately I turned up with a pair of goalkeeper gloves, one of the others on the team decided I must be a goalkeeper then. And that’s the position I played for many years, and I loved it.

Now I’ve been a backend developer for many many yeas and I love that as well. For me there are some obvious parallels between the two roles.

Even a small mistake is a big mistake

I clearly remember charging out of goal to meet the incoming ball and blast back down the pitch to start an attack. Instead it bounced higher than I anticipated and went over my head. For any other player on the pitch and this wouldn’t have mattered but for me it meant we conceded a goal and were now losing.

As a backend developer a tiny mistake such as a missing semi colon can bring a whole site down. Compare that to front end where missing out whole closing tags can often be rectified by the browser. Those front end developers don’t know how easy they have it, is that why we make them learn a million different JavaScript frameworks and throw in inconsistent box models in browsers.

Leading from the back

One of my favourite parts of football was corners. I’m very average height but I can jump a lot higher than most people. I used to love to come running out, jumping through the air with a loud shout and claiming the ball. It let everyone in the team know I was there and taking care of my job.

Providing a solid back end for others in your team is critical to building successful websites and applications. From mapping out database architecture to providing valuable end points, your work is what feeds the output and allows for the rich content we have come to expect today.

I always evangelise the value of having a good base to build on. If we can have the data in a good format to begin with then pulling it out again in desired formats and combinations becomes a lot simpler. The phrase “S**t in, s**t out” is very appropriate here.

Embrace your personality type

Every year when we ordered a new team kit I was handed the brochure to make my choice. I flicked through and stopped on the one that was the brightest and loudest. I’ve heard many others, including professionals, talk about goalkeepers having a certain personality type and I could definitely relate.

I think back end developers also have a clear personality type. We are very analytical and love a problem. There are plenty of times I see my colleagues design work and think “I wish I had the design eye they have”, but I don’t, and I encourage you to embrace your place in web development that is such a wonderful ecosystem of so many different types of people.

A closing note

After the ball bouncing over my head incident I cried on the way home. In my first developer role I deleted a colleagues database they’d spent all morning crafting. I didn’t cry but did feel very very bad and spent the rest of the day helping them restore it. Individual mistakes happen but with a good team around you those mistakes will become tiny blips.

Make working from home work for you and your team

When I started working for Headscape I also started working from home. Our office is a 3 hour drive form where I live so commuting isn’t an option and as a family we were not looking to relocate.

In the past few years I’ve learnt a lot from home working and how to make it work for me and our team. Here are some tips that I have found smooth the process.

Be Sociable

We’ve used various chat tools to communicate over the years and I always ensure that I’m signed in and active.

A simple “Good morning” to everyone lets your team know you are at work and instantly opens up the opportunity for any dialogue. I like to think of it in the same way you would see someone is at their desk in a shared office.

On the flip side of this if you need to be left alone to really dig into something just let everyone know and either turn off communication tools or set yourself to away.

Be Consistent

It helps that I have a young family so keeping consistent work hours is good for family life. I find it’s also good for work life balance. I have clear start and end times and this stops work bleeding over into my outside work time.

I’m sure it would be very easy to start a bit late, have a longer lunch, then keep working into the evening. This could easily feel like you had actually worked all day long.

Be Available

Similar to being sociable I, nearly, always like to have some form of communication open. A quick response in Slack or a 5 minute chat on Skype can really help out another member of the team.

This is the sort of thing that can help a team be at it’s most productive and reduce any negative impacts of not being in the same physical space.

Be Grateful

Working from home is a great benefit for me and I feel very lucky that I have a job that can so easily allow me this privilege. It’s very easy to get caught up in the day to day of work, but every so often I remember how nice it is that I have that extra time each day saved by not commuting and the fact I’m there to see my children as they head off to school and when they return home.

Be Whatever

These are my experiences from the past few years and I appreciate everyone has different ways of working. Overall I would encourage individuals or companies to try home working and make it work for you.