Skip to main content

How to add search to a website using flexsearch

·2373 words·12 mins
Hugo
pdyc
Author
pdyc
Table of Contents

TLDR; Describes how to implement local search using flexsearch js library to website

I added documentation to Easyanalytics, and one of the things that I had to do was to add search functionality to the documentation. Later, I forgot the exact method I used to implement it 😀. You can read more about it here. So, I decided to write an article on the process, which would also serve as guide to the documentation. The scope of the article is local search, specifically implementing it with the FlexSearch library. I have provided references to other interesting search libraries at the end, in case you want to explore further.

Why do we need website search

Adding search functionality to your website can greatly enhance the user experience by making it easier for visitors to find the information they're looking for. Without search, users may have to navigate through multiple pages or rely on site navigation, which can be frustrating, especially on larger websites. Search allows users to quickly find relevant content by entering keywords or phrases.

This is especially true if you are creating documentation for a software product or api. Simply because all the information is not useful to the user, often user wants to accomplish something and is looking for specific information and wants to get there as quickly as possible there is no time for reading the entire documentation.

Approaches to website search and choosing the right approach

When it comes to adding search functionality to your website, you have several options to choose from. You can opt for commercial solutions or open-source alternatives, including local search that does not involve any server or server-based search, as well as solutions tailored for static websites or dynamic content management systems (CMS).

Server Based Vs Local Search Solutions:

Local search solutions, such as client-side JavaScript libraries, execute the search on the user's browser, offering a faster and more responsive experience. However, they may have limitations in terms of indexing large amounts of data or handling complex search queries. On the other hand, server-based search solutions, like those provided by popular search engines such as Elasticsearch or Algolia, can handle large datasets and offer more advanced search capabilities, albeit potentially coming with additional costs or infrastructure requirements. Here are some scenarios where local search shines:

  • Privacy Concerns: Local search offers a solution by keeping user data within their device. Since search queries are performed locally, there's no need to transmit sensitive information to external servers, ensuring greater privacy and control over personal data.
  • Offline Usage: Local search is invaluable for applications or websites that need to function seamlessly offline. By performing searches within the user's browser or device, search functionality remains accessible even when there's no internet connection available. This is particularly beneficial for progressive web apps (PWAs) or offline documentation systems.
  • Static Sites with No Servers: For static websites that don't rely on server-side processing or database queries, local search provides a lightweight and efficient solution. Content can be pre-indexed and packaged with the website, enabling users to search through the site's content without the need for server infrastructure. This simplifies deployment and reduces hosting costs for static site owners.

For bigger sites and complex search options, it is better to go with server based solution. This is because adding local search involves downloading a large amount of data to the user's device, which can be computationally expensive and may slow down the user's device. If your site has thousand plus pages and you want to enable search on all of them, a server-side solution would be a better fit. Additionally, server-side solutions provide better support for complex search requirements, such as multi-language support, relevance ranking, and filtering options.

Using Local Search Libraries:

Using local search libraries, such as FlexSearch, involves following steps:

  1. Creating Index: The first step is to create an index of the content you want to make searchable. This typically involves extracting relevant text data from your website or application, such as titles, descriptions, summaries, or full content, depending on your specific use case.
  2. Adding Index to the Library: Once you have extracted the necessary text data, you need to add it to the search library. This enables the library to efficiently search through the content and retrieve relevant results based on user queries.
  3. Searching the data: Third step is to search the data with user query.This typically involves creating a search function that takes user input and performs the search against the index.
  4. Displaying results: Final step is to display results. To make the search functionality accessible to users, you'll need to create a user interface (UI) that allows users to enter their search queries and displays the search results. This can be done using HTML, CSS, and JavaScript, and can be integrated into your existing website design. In your website's HTML, you'll need to create a search input field and a container for displaying the results. Then, using JavaScript, you can capture user input, trigger searches, and dynamically display the search results.

Lets take an example with Flexsearch to make it more clear. FlexSearch is a lightweight and fast client-side search library written in JavaScript. It is designed to provide efficient full-text search capabilities for web applications, including static websites.

Compared to other client-side search libraries, FlexSearch offers superior performance, especially when dealing with large datasets or complex search queries. It supports advanced features such as phonetic search, relevance scoring, and pagination, making it a versatile solution for a wide range of use cases.

While the example provided focuses on FlexSearch, it's worth noting that other local libraries generally follow similar steps with minor differences in their APIs. The fundamental process of creating an index, adding data, searching, and displaying results remains consistent across these libraries.

Example Using FlexSearch:

Here's a brief example demonstrating how to use FlexSearch for local search:

    // Step 1: Example data this needs to be extracted from website in this form final result is shown here
    const documents = [
        { id: '1', title: 'Sample Title 1', description: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit.' },
        { id: '2', title: 'Sample Title 2', description: 'Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas.' },
        // Add more documents as needed
    ];

    // Step 2: Adding index/document to the library
    var index = new FlexSearch.Document({
        tokenize: "forward",
        cache: 100,
        document: {
            id: 'id',
            store: [
                "title", "description"
            ],
            index: ["title", "description"]
        }
    });

    documents.forEach(item => {
        index.add({
            id: item.id,
            title: item.title,
            description: item.description
        });
    });

    // Step 3: Searching
    const query = 'sample'; // Example search query
    const results = index.search(query);

    // Step 4: Display search results
    console.log(results);

In this example, we create an index using FlexSearch and add sample documents containing titles and description. We then perform a search using a sample query and display the results. This demonstrates the basic usage of FlexSearch for implementing local search functionality.

Incorporating FlexSearch in Static Site Generators: Hugo

Now let's see how to create document index for a website. Since this part lies outside the scope of Flexsearch library we skipped it in the previous Flexsearch example.

Hugo is a widely-used static site generator known for its flexibility and ease of use. If you're using Hugo, you can take advantage of its data templates and built-in functions to create a search index and integrate FlexSearch into your website. Hugo's data templates enable you to generate JSON files containing your website's content. By creating a template that extracts the required data, such as page titles and summaries, you can easily generate a JSON file for FlexSearch indexing. Here's a step-by-step guide on how to integrate flexsearch with hugo

  • Creating a Data Template for Indexing: First, create a data template file (e.g., `layouts/_default/search.json`) that generates the search index based on your content. Here's an example that indexes the title and description of your posts:

       {{ $list := slice }}
       {{- $list = (where .Site.RegularPages.ByTitle "Section" .Section) }}
       {{ $currentSection := .CurrentSection }}
       {{ $jsonArray := slice }}
       {{ range $index, $element := $list -}}
       {{ if in (.RelPermalink | string) $currentSection.RelPermalink }}
       {{ if not .Params.exclude }}
    
       {{ $jsonObject := dict "id" $index "href" .RelPermalink "title" (.Title | jsonify) }}
       {{ with .Description -}}
       {{ $jsonObject = merge $jsonObject (dict "description" (. | jsonify)) }}
       {{ else -}}
       {{ $jsonObject = merge $jsonObject (dict "description" (.Summary | plainify | jsonify)) }}
       {{ end -}}
    
       {{ $jsonArray = $jsonArray | append $jsonObject }}
       {{ end }}
       {{ end }}
       {{ end -}}
       {{ $jsonArray | jsonify }}

    This template iterates over your site's regular pages, creates a dictionary with the title, description, and permalink for each page, and appends it to the `jsonArray` variable. Finally, it converts the `jsonArray` variable to JSON

  • Using the Data Template in Content Section: Next, you will need to incorporate this data template into your content section to generate the index. Create a search.md file in section where you want to enable search. For instance if you want to enable it on docs than add it to the `content/docs/` folder

    +++
    layout = "search"
    outputs = ['json']
    exclude = true
    +++
  • Adding FlexSearch JavaScript Library and Loading Index Data: Download the FlexSearch library and add it to your Hugo project's static folder, for instance, static/js/flexsearch.js. Then, include the library in your HTML template. Create a JavaScript file that includes the FlexSearch library and initializes the search functionality.

    Utilize Hugo's built-in asset pipeline or a custom script to include the FlexSearch JavaScript file in your website.

    <script src="/js/flexsearch.js"></script>

    Next, load the index data into FlexSearch and set up the search functionality. Here's how you can modify the code to achieve this:

          <script>
          // Initialize FlexSearch library
      var index = new FlexSearch.Document({
          tokenize: "forward",
          cache: 100,
          document: {
              id: 'id',
              store: [
                  "href", "title", "description"
              ],
              index: ["title", "description"]
          }
      });
      // Load index data asynchronously
      // dataUrl is the url that contains search.json
      fetch(dataUrl)
          .then(response => response.json())
          .then(data => {
              // Loop through the data and add it to the index
              data.forEach(item => {
                  index.add({
                      id: item.id,
                      href: item.href,
                      title: item.title,
                      description: item.description
                  });
              });
          })
          .catch(error => {
              console.error('Error fetching data:', error);
          });
      </script>

In this code:

  • We use the `fetch` API to asynchronously load the index data (`search.json`) from the server.
  • Once the data is loaded, we initialize a FlexSearch and add the index data to it.

Querying and Displaying results

  <!--- //sample code for setting up user input and display result in html--->
  <input type="text" id="searchInput" placeholder="Search...">
  <ul id="searchResults"></ul>

  <script src="/js/flexsearch.js"></script>
  <script>
    // JavaScript code snippets go here
  </script>
  • This is sample code to create a search input field and results container.
  // sample code for querying and displaying result
  var inputElement = document.getElementById('searchInput');
  var resultsElement = document.getElementById('searchResults');

  inputElement.addEventListener('input', function() {
      var query = inputElement.value;
      var results = index.search(query);

      // Display search results
      resultsElement.innerHTML = '';
      results.forEach(function(result) {
          var li = document.createElement('li');
          li.textContent = result.title;
          resultsElement.appendChild(li);
      });
  });
  • This is sample code tp use JavaScript to capture user input, trigger searches using FlexSearch, and dynamically update the search results.

With these steps, you can integrate FlexSearch for local search functionality in your Hugo static site.

Tradeoffs in Indexing:

When creating the index, you will need to consider tradeoffs in terms of the amount of data included. Using the entire content of your website or application for indexing provides the most comprehensive search results but will increase the size of the index and require more resources for processing and storage.

On the other hand, indexing only specific parts of the content, such as titles, descriptions, or summaries, reduces the size of the index and resource requirements. However, this approach may result in less comprehensive search results, especially if users are searching for specific terms or phrases contained within the full content.

Strategies for Smart Indexing:

To optimize indexing for local search, consider strategies such as:

  • Prioritizing important content: Indexing key sections or pages of your website or application that are frequently searched or contain valuable information. For example, only enable searching for documentation section of your product.
  • Using metadata: Leveraging metadata or structured data to identify and prioritize relevant content for indexing, such as tags, categories, or keywords.

Other search libraries

Here is the list of other search libraries for reference

Category Solution Description
Open-Source Server-Based Solutions Elasticsearch Highly scalable search engine based on Apache Lucene, offering full-text search, real-time indexing, and analytics features.
Apache Solr Robust search platform with advanced features like faceted navigation and distributed indexing, customizable for enterprise environments.
MeiliSearch Fast and easy-to-use search engine built with Rust, offering typo tolerance, faceted search, and real-time indexing.
Vespa Big data processing and serving engine developed by Yahoo, providing powerful search and recommendation capabilities with complex ranking algorithms.
Commercial Server-Based Solutions Algolia Hosted search API with fast and relevant results, offering features like typo tolerance and real-time indexing, available with various pricing plans.
Amazon Elasticsearch Service Fully managed Elasticsearch service by AWS, providing scalability, reliability, and easy integration with other AWS services.
Swiftype Hosted search solution by Elastic, offering powerful search capabilities, intuitive interface customization, and easy integration with popular CMS and e-commerce platforms.
Local Search Solutions Fuse.js Lightweight JavaScript library for fuzzy search, providing advanced features like fuzzy matching and exact phrase matching.
Lunr Lightweight JavaScript library for full-text search in the browser, suitable for small to medium-sized datasets without the need for server-side processing.
MiniSearch Another lightweight JavaScript library for client-side search, optimized for speed and efficiency, offering features like fuzzy search and relevance ranking.
FlexSearch Fast and memory-efficient full-text search library in JavaScript, suitable for both client-side and server-side implementations, with support for advanced features.
Pagefind Another JavaScript library for full-text search in the browser.

Conclusion

Flexsearch is a great library that can be used to add local search to your website. I am planning to change the theme of Easyanalytics so my effort of implementing Flexsearch based search is likely to go waste but it was good to learn how local search is implemented. There are some more use cases like searching for ai prompts so you don't have to keep typing them again and searching data from your bookmarks before searching on internet where this kind of approach can be useful.

Related

How to use org mode in Hugo
·967 words·5 mins
Hugo
Did you know that Hugo supports formats other than markdown for content?
Migrating existing page to hugo static site generator
·1227 words·6 mins
Hugo
TLDR: How to migrate existing site to Hugo
Video Editor with FFmpeg Commands
·1041 words·5 mins
Video Ffmpeg
Easily edit videos, add logos, text and export customized FFmpeg commands for advanced video editing workflows.
Million dollars is not cool, you know what is cool? million rows
·1922 words·10 mins
Csv React Table Js
TLDR: Story of challenges faced while displaying large csv file with million+ rows in CSV Viewer
CSV Viewer with charts
··1394 words·7 mins
Csv Chart Tool
CSV Viewer with charts, supports viewing CSV, sorting, searching, filtering of data, plotting charts, setting title and saving charts as images.
All in one dashboard for google analytics and search console plan & execution
··2265 words·11 mins
Indiehacker
This page lists project plan and execution details of all in one dashboard for stats on google properties