Ian Luckraft Web Development | Web Design Plymouth

When to use a custom search solution

This post also appears on the Headscape site.

Website search can often be a thorny subject. Expectations of search capabilities have been set very high by search engines. The default search offered by content management systems (CMSs) is often a bit basic by comparison. For example, the results are often not the most relevant to the search term used because the CMS is only offering a simple “does this page contain this word” method.

Our discovery work for the Competition Appeal Tribunal site revealed a specialised set of search requirements. When working with a CMS, I’m a firm believer that you go with the flow of its strong and weak points, and try not to fight it. With that philosophy in mind, my instinct was not to struggle to make native CMS search fit the requirements but rather to develop a custom solution for this site.

We opted for Elasticsearch, which is described as a “RESTful search and analytics engine”, combined with the very well maintained and documented Elasticsearch PHP library.

As soon as you put some data into Elasticsearch it does a good job of returning relevant results. But where it really excels is when you begin to fine tune it to your needs.

As an example, there is a field on the site that we needed to be able to perform full text searches and sorts. This can be accomplished by duplicating the field into two different “types” allowing one to have all of the full text search capabilities whilst the other is used for sorting.

On another project for US law firm Buckley Sandler (that is built on Drupal), the different content types and fields did not all have the same relevance in search results. By applying a “boost” to specific content types and fields we were able to deliver the most relevant results to users. Whilst Drupal’s native search API does allow a boost to be applied to certain fields, using Elasticsearch we were able to boost specific content types, individual pages and taxonomy terms.

This level of customisation may not be required on your project, but if it is I highly recommend trying Elasticsearch as I’m confident it can meet your needs.

Chatbots, always available customer support or know nothings?

This post also appears on the Headscape site.

After reading Build Chatbots with PHP by Christoph Rumpel I started making some simple chatbots and quite quickly moved on to more complex interactions. It didn’t take long to get something basic up and running, which in turn, got me really excited about the possibilities.

But – there’s always a ‘but’ isn’t there – it also didn’t take long for the bots to fail to have answers, especially when I showed them to colleagues who didn’t know the precise phrases the bots were listening for.

To try and get over this I used Dialogflow for the backend as it has natural language parsing capabilities. It also has a very nice UI that will allow even non-technical users to build up conversations. After a short tutorial I found Dialogflow easy to use and very powerful for the little effort needed to start using it.

Research

Highlighted by others asking my bot questions it did not understand, it is essential that you do in-depth user research before you launch your bot. The last thing you want is to spend a lot of time and money only for people to never use the chatbot as they had a bad first experience.

Just like any web project, speaking to the people who will be interacting with your site / bot will show you interactions you can never have imagined by yourself.

Monitor

It’s highly likely that you run some analytics or monitoring code on your website to ensure users are moving through the site as expected. You should also be monitoring the chatbot’s interactions to see where users take conversations and if any of these lead to dead ends.

This could be in the form of capturing the sessions with a tool such as Full Story, or logging the conversations to a database that you can then analyse at a later date.

Learn

Machine learning is great but machines also need help as well. Whilst Dialogflow has a machine learning aspect to it there will more than likely be cases where a user asks a question that it cannot learn from. By spotting new forms of interaction early you can return to the research phase to see how others would phrase the same question and then feed these alternatives into the backend.

You may also find people are looking for specific areas of your site and your overall information architecture could do with some readjustment. Or better still, maybe people are coming to you looking for a service or product you do not yet offer but could do, leading to new lines of business.

Chatbots are an exciting development and give the potential to have a form of customer support that’s always on hand. But they are not something that can be set up and never looked at again, otherwise their lack of knowledge can leave users feeling frustrated and abandoning your site.

Blogging, Completed It Mate

Oddly enough I’d had this post in mind over the summer and then some gentle encouragement from Paul Boag on an episode of The Boagworld Podcast convinced me to finish writing it.

Towards the end of last year I had several ideas for blog posts rattling round my head. I made a conscious decision I should actually finish the blog posts instead of leaving them as “Draft” and thinking “I’ll finish that soon”.

Off To A Flyer

When you have lots of ideas the posts just flow, you have momentum and enthusiasm and you’re a blogger master.

I even started setting myself reminders and began blogging quite regularly. Then it stopped. Like so many people stop and I’m not sure why. I missed a week, promised I’d finish the post next week and then all of sudden I was out of the habit.

Why Blog

I had a think about what made me start blogging in the first place

Had some ideas
Self documentation
Sure up my own thoughts and processes
Help others

It was good to finally finish off some of the blog posts I had stored as drafts and that made me look at other aspects of my job and think “I could post about that”.

I’m sure most developers document parts of their jobs that they don’t do so frequently. My own website seems a good place to keep those documents.

A bit like rubber ducking the act of writing out a process really helps me to straighten out my thoughts and ensure I’ve covered all aspects of a job.

It’s been really rewarding to see links coming in from drupal.org and seeing some search terms people have used and ended up on my blog such as Testing SSL certificates without a domain name.

When to Blog

Maybe I aimed a little too high with once a week at the start and things started to feel forced.

My aim going forward is to commit to once a month but if I have a series of posts I’ll possibly look at releasing those a little closer together.

Ultimately I want it to be a thing I enjoy rather than forcing myself to do it.

Thanks To Anyone Who Blogs

I’ve learnt so much from blog posts over the years. People just sharing a little bit of knowledge they’ve learnt to people consciously documenting the process they’ve been through.

It’s also given me a huge appreciation of people like Paul Boag who have consistently produced lots of high quality content over a long period of time.

Maybe, just maybe, this blog post will make you write a blog post, and you never know maybe I’ll find that blog post and it will help me. Even if that never happens there is a good chance it will help someone, even if it’s just your getting your thoughts in order.

Today I am Not Working From Home

I’m a full time remote worker and have a room at home set up as an office I can shut myself away in. As I’ve talked about before I love working from home but this week for the first time I decided to try remote working from a public space. I choose a local Wetherspoons as it seemed to tick all the boxes and was a short walk from home.

Setting Off

The night before I actually made a list to ensure I’d have everything I needed. Small things like earphones that I don’t normally use should come in very handy in a public space. One last mental check before leaving home and I was off. Luckily it was a dry morning and having a walk first thing was very nice.

Essentials, Free WiFi and Power

A lot of places have free WiFi these days so I knew that wouldn’t be a problem but it was only on my way that I realised I would need a power supply to last the whole day, rookie error. I opted for the honest approach and just asked when I arrived. Handily there was a nice table tucked away at the back with a power socket.

What do I do When I Need the toilet?

Everything was going well and then I hit my first “What do I do now?” moment. When I need to pop to the toilet what do I do with all my stuff. By this time there was a table of retired gentlemen sat on the table beside me. I had protectors of my belongings.

Not as Many Screen Breaks

When I’m at home I find myself getting up to make a drink a bit more often and normally have few paces around the kitchen while I wait for the kettle to boil. Being somewhere where food and drink are brought to you I found myself sitting in the same place for a lot longer than I usually would.

Planning Calls

As I mentioned in my “Make working from home work for you and your team” post I like to make myself available for quick chats with colleagues. Today I had no client calls planned and didn’t end up needing to chat with anyone else. I had my earphones and I suppose I would just have had to try and talk at a polite volume had I needed to.

How Long Can I Stay

Then came the second “What do I do now?” moment. When I arrived I ordered food and drink, but now I feel like I’ve outstayed my welcome, and that was nothing to do with the staff just my internal thoughts. So I ordered another drink and a small bite to eat.

Overall Productivity

When lunchtime came I made sure to leave and have some time away from the “desk” just as I do at home. I got plenty done away from home and don’t feel it impacted my productivity at all. It was nice to be somewhere different and it adds to my enjoyment that I’m lucky enough to have a job where I can do such a thing.

Would I do it Again?

Yes, but certainly not everyday. Ignoring the cost of buying food and drink so I feel like I can stay, I do like being in the home office. Maybe I shall save it as a once a month treat.

Fractal – A Backend Developers Perspective

One of the great thing about attending conferences is attending a talk and learning about something you didn’t even realise you wanted to know about.

During DrupalCon Vienna I saw a talk by Anton Staroverov and Tassilo Groeper from Wondrous titled Decouple your Twig from PHP and make Frontenders happy! In the talk they showed a tool called Fractal which has been developed by Clearleft. After seeing the presentation I felt this would be a very useful tool so set about looking into it further.

Easy to Use

That evening I started to try out Fractal and within 10 minutes I had a build up and running and had added a component to it.

Since then at Headscape we have switched the default templating language to Twig to tie it into our Drupal 8 development. We’ve also used it’s static output to produce pattern libraries for clients.

Reference Point for All

Not only is Fractal a useful tool for frontend developers as mentioned above we also use it to show clients both individual components and components working together.

As a predominantly backend developer I also find it useful when I need to add a button for example. I can easily find all the button options in the nice frontend UI and see what classes I need to add.

Helps Building Templates and Reusable Components

Having one place where we can list all of the components and then using this as the actual templates for the production build is very useful for me.

During development, wether that is early stages or updates in the future, I find having a reference point of components speeds up the process.

For example if we have a new content type I can look at the various view modes we already have to see how I should be outputting the content.

With how quick it is to get something up and running with Fractal I’d encourage you to try it out. It might take a little bit of settings adjustment and maybe even splitting you’re templates and CSS out into more componentised methods but I think the benefits you’ll gain will be worth it.

The Importance of Real Data

At the start of a project there are many different objectives and everyone involved has their own sense of priorities or tasks they wish to a tackle first. For me that is generally sorting the backend into a nicely organised structure that will allow flexibility in the output both now and in the future.

A big part of this is getting real data in so the front end output is as close to final content as possible. This allows us to realistically analysis how the content and design will work together.

Does it Fit

As flexible as the web is with it’s layouts there are still times when a design will only allow a certain amount of text or number of items to appear before it looks squeezed. For example text in a sidebar can be excellent if it’s short and snappy. However if the actual content is a long paragraph it will be very hard to read.

I’ve also experienced this with titles where they can be in a large font size. Taking into consideration how that works over multiple lines is a useful exercise.

Lining Up

Adding columns or boxes to a layout is a useful way of defining content areas. Without real data these areas normally contain filler text, occasionally the same filler text, which can mean boxes are the same height as each other.

On many occasions I’ve seen a design change as soon as real content is output as one boxes contains many more lines than the other causing them to no longer vertical match in height.

There are ways to handle this with JavaScript and more and more CSS but it is still a consideration at this point in time.

Missing Anything

This is normally the realm of imagery as it is the hardest part of content to populate. Instead of just using test images take the time to ensure the design works with other no images or a placeholder repeated multiple times.

CMS Tools

Content editors often need to add some classes or styles to text entered in wysiwyg editors to help front end presentation. By their nature these editors don’t or can’t always output perfect markup. By using them to populate the site at design and development time you can save a lot of headaches by utilising the markup they do produce and not battling against it.

Overall

On every project I find the use of real data and content smooths out the development process as all involved parties can see a “real world” example of how the site will function with it’s content.

Drupal 8 Workflow Notifications with Rules Part 3

In previous posts I’ve covered enabling and editing a Workflow and then sending an email when a new content moderation state has been saved.

In this post we will complete the loop by sending emails when a content moderation state has been updated.

In this example an editor will publish the node and the rule will email the node creator to tell them it has been published.

Head to the Rules admin page /admin/config/workflow/rules and click Add reaction rule

This time we want to choose After updating content moderation state from the React on event select.

This time round we will need two conditions.

Check the unchanged moderation state
Check the new moderation state

Click the Add condition button and then choose Data comparison from the select.

For the data to compare value you can either use the data selection or enter

content_moderation_state_unchanged.moderation_state.value

Then for the data value enter review

This is the moderation state the node was in before it was saved.

Now click save.

Add another data comparison condition then this time enter the following

Data to compare value

content_moderation_state.moderation_state.value

Data value

published

Next we need to fetch the entity so we can use it’s values in our email.

Back on the edit page for this rule click Add action and choose Fetch entity by id from the select.

For Entity type enter node and for the Identifier value enter

content_moderation_state.content_entity_id.value

Then click save.

Now we can send an email to the node creator.
Click Add action and choose Send email.

In the Send to field use direct input and enter

This is a token that gets replaced with the users email address.

For subject we can just enter something appropriate such as Published Notification

For the message we will use the direct input mode an a token that provides the entity title

Your content {{entity_fetched.title.value}} has been published.

Save that and then save the rule.

As is often the case with Drupal it’s worth clearing your cache before testing.

Rules is still in it’s infancy for Drupal 8 but I feel it provides enough functionality and is well enough supported that it can be included in production site.

Changing the Default Content Moderation State in Drupal 8

I have been working with the Content Moderation module in Drupal 8 and one slight niggle we came across was the default option for the Moderation state.

If the current state was In Review we would like the Change to option to also be In Review rather than Draft.

This means the default is to keep the entity in it’s current state rather than switching to a new state. Switching to a new state is then an active task for the editor.

I did this by implementing hook_form_alter() in a custom module.

/**
 * Implements hook_form_alter().
 */
function my_module_form_alter(&$form, \Drupal\Core\Form\FormStateInterface $form_state, $form_id) {

if ( array_key_exists('moderation_state', $form) ) {

$node = $form_state->getFormObject()->getEntity();

$moderationState = $node->get('moderation_state')->getValue('state');

if (isset($moderationState[0]['value'])) {
 $form['moderation_state']['widget'][0]['state']['#default_value'] = $moderationState[0]['value'];
 }

}

}

In this function we first check if the form contains the moderation_state key.

Next we load the node so we can access it’s properties.

The moderation state is then retrieved from the node.

The last part is to check if the moderation state has a value. If we are creating a node this will not be set.

In the if statement we then set the default value to the current state.

Drupal 8 Workflow Notifications with Rules Part 2

In my previous post I talked through setting up Content Moderation with a review state. In this post we will complete the process by sending email notifications via the Rules module when the moderation state is changed.

Install Rules

To enable the Rules module we first need to require it via Composer. Run the following from the command line in the root of your project
composer require drupal/rules

Once this has run you can return to the Extend page and enable the Rules module.

Create the First Rule

By the end we will have three rules for different states and transitions. The first rule will handle the creation of content the state is set straight to Review.

To create a rule navigate to Manage -> Configuration -> Rules (/admin/config/workflow/rules)

Once on that page click on the Add a reaction rule button. Fill in the details with the following.

Label: Notify reviewers of content created for review

React on event: After saving new content moderation state

We now need to add a condition so click the Add condition button.

From the select box choose Data Comparison from under Data and then click Continue

When adding our condition it is useful to Switch to data selection for the Data selector input. This will help us drill down into the available data.

The Data selector we want is content_moderation_state.moderation_state.value

Leave the Operator as ==

Set the Data value to review. This is the machine name of our state that was created in the Editorial Workflow.

We can now save our condition.

Adding an Action

Before we create our action to send the email we need to make the node available to our rule so we can use it’s title and ID in the email body.

Click Add action and then choose Fetch entity by id under the Entity section.

On the next screen set the Entity Type value to node and then for the Identifier we will again switch to data selection.

Using the autocomplete we set the Identifier value to content_moderation_state.content_entity_id.value

Sending the email

We can now add an action for sending the email. Click the Add action button and choose Send email from the System section of the dropdown.

For the Send To we use an email address that will forward to all reviewers, e.g. reviewers@example.com

For the Subject we use Item Ready For Review.

For the body we want to provide some information for our reviewers so we set it to

Please review {{ entity_fetched.title.value }} at https://example.com/node/{{ entity_fetched.nid.value }}

The two parts in curly brackets are tokens from our fetched entity, node, that we performed earlier.

When the email sends the body would would look something like

Please review My New Page at https://example.com/node/12

In the next part we will create two more rules to handle updating the moderation state.

Drupal 8 Workflow Notifications with Rules Part 1

For Drupal projects I’ve always used the comprehensive suite of tools provided by Workbench. But when I started a new project and saw that Workflow is hopefully making it’s way into core it felt like a good time to try it out. One part of the project was the addition of notifications when Workflow states were changed. With the Rules module also receiving lots of attention I decided combining the two modules was the best approach.

The following example was made using Drupal 8.4.4

Enable the Modules

First of all we have to enable the Workflows module and the Content Moderation module. Click on Extend form the admin menu or go to /admin/modules and find these two modules.

When enabling Content Moderation you will be warned that it is an experimental module. https://www.drupal.org/core/experimental

Edit the Workflow

If you go to /admin/config/workflow/workflows you will see we have one workflow called Editorial.

Click the Edit button to go to /admin/config/workflow/workflows/manage/editorial

We currently have three states but we want to add a new one for Review. Click the Add a new state link. On the Add state page all we need to enter is a label of Review and then click Save.

Back on the Edit Editorial Workflow page drag the Review state to in between Draft and Published and then click the Save button at the bottom of the page.

We now need to edit the transitions to allow the flow between draft, review and publish. This is how the transitions look by default. We need to enable a draft to be become a review and a review to become published. We want to cut out the draft to published transition.

Below are the default transitions, we’ll first add our new ones and then modify the existing ones.

Click the Add a new transition link and you will be on a page with options for Label, From and To. Set the Label to Review, From to Draft and To as Review then click Save.

Back on the main page drag the new Review transition up so it is between Create New Draft and Publish and then click Save.

We now need to update the existing transitions. Change the Create New Draft From value. It is currently Draft, Published and we need it to be Draft, Review.

Lastly for transitions change the Publish transitions’s From value to Review, Publish. This means a node has to go from Draft to Review before it can be published.

Add The Workflow to Entities

The last part of the workflow setup is to enable it for any required content types or entities. At the bottom of the page there is the section This Workflow Applies To. In this example we will enable this workflow for Basic Pages.

Click the Select button beside Content types, select Basic Page from the overlay and then click Save. Now If you go to add a Basic Page you will see at the bottom there is a select where you can choose the State of you node.

In the next part we will look at sending email notifications when the Workflow state changes using the Rules module.