Machine Learning in Frontend : Context Menu Position with Brain.js and React

A learning exercise to implement a neural network for context menu position prediction using Brain.js and React

Introduction

In web development, context menus provide a list of actions or options that are relevant to a specific element or area the user interacts with. One common issue developers face is ensuring that context menus appear within the viewport, especially when triggered near the edge of the screen. While this problem can be effectively addressed using standard programming techniques, we will explore an alternative approach for learning purposes: using a neural network to predict the position of the context menu. In this article, we will implement a simple neural network for this task using the Brain.js library and integrate it into a React application.

Table of Contents

1. Introduction to Brain.js
2. Preparing the Dataset
3. Creating and Training the Neural Network
4. Building the React Application
5. Integrating the Model into the React Application
6. Conclusion

Introduction to Brain.js

Brain.js is a JavaScript library for implementing neural networks in the browser and Node.js. It provides a simple API for creating, training, and using neural networks for various tasks. In this article, we will use Brain.js to create a feedforward neural network that predicts context menu positions based on mouse coordinates and viewport dimensions, and we will integrate this neural network into a React application.

Section 2: Preparing the Dataset

To train our neural network, we need a labeled dataset containing various combinations of input features and corresponding output labels. The input features include mouse click coordinates, viewport dimensions, and context menu dimensions, while the output labels represent the desired coordinates for the context menu position. You can create this dataset by simulating different scenarios or by collecting data from real users.

// src/utils/trainModel.js

const rawData = [
  // When close to top left corner
  {
    input: {
      clickX: 10,
      clickY: 10,
      viewportWidth: 1000,
      viewportHeight: 800,
      menuWidth: 200,
      menuHeight: 160
    },
    output: { x: 10, y: 10 }
  },
  // When close to top right corner
  {
    input: {
      clickX: 900,
      clickY: 10,
      viewportWidth: 1000,
      viewportHeight: 800,
      menuWidth: 200,
      menuHeight: 160
    },
    output: { x: 700, y: 10 }
  },
  // When close to bottom left corner
  {
    input: {
      clickX: 10,
      clickY: 720,
      viewportWidth: 1000,
      viewportHeight: 800,
      menuWidth: 200,
      menuHeight: 160
    },
    // move the context menu so that it is within viewport
    output: { x: 10, y: 560 }
  },
  // When close to bottom right corner
  {
    input: {
      clickX: 900,
      clickY: 720,
      viewportWidth: 1000,
      viewportHeight: 800,
      menuWidth: 200,
      menuHeight: 160
    },
    // move the context menu so that it is within viewport
    output: { x: 700, y: 560 }
  }
  // Add more data points for better prediction
];

We need to ensure that the input and output values are normalized to a range between 0 and 1. For normalizing data, we will create a utility function that takes the raw data and normalizes the input and output values between 0 and 1. Similarly, we will create another utility function to denormalize the predicted values back to their original scale

// src/utils/normalizeData.js

/**
 * Utility function to normalize data
 */
export const normalizeData = (dataPoint) => {
  const input = dataPoint.input;
  const output = dataPoint.output;

  return {
    input: {
      clickX: input.clickX / input.viewportWidth,
      clickY: input.clickY / input.viewportHeight,
      viewportWidth: 1,
      viewportHeight: 1,
      menuWidth: input.menuWidth / input.viewportWidth,
      menuHeight: input.menuHeight / input.viewportHeight
    },
    output: {
      x: output.x / input.viewportWidth,
      y: output.y / input.viewportHeight
    }
  };
};

// src/utils/denormalizePrediction.js

/**
 * This function will transform prediction 
 * from the neural net to the x and y position 
 * of the context menu
 */
export const denormalizePrediction = (prediction, viewportWidth, viewportHeight) => {
  return {
    x: prediction.x * viewportWidth,
    y: prediction.y * viewportHeight,
  };
};

Creating and Training the Neural Network

Using Brain.js, we will create a feedforward neural network with one or more hidden layers. Configure the network options, such as the number of iterations, error threshold, and logging, then train the network using the prepared dataset.

// src/utils/neuralNetwork.js

import brain from "brain.js";

export const neuralNetwork = new brain.NeuralNetwork();

// src/utils/trainModel.js

import { normalizeData } from "./utils/normailzeData";
import { neuralNetwork } from "./neuralNetwork";

const rawData = [
  // When close to top left corner
  {
    input: {
      clickX: 10,
      clickY: 10,
      viewportWidth: 1000,
      viewportHeight: 800,
      menuWidth: 200,
      menuHeight: 160
    },
    output: { x: 10, y: 10 }
  },
  // When close to top right corner
  {
    input: {
      clickX: 900,
      clickY: 10,
      viewportWidth: 1000,
      viewportHeight: 800,
      menuWidth: 200,
      menuHeight: 160
    },
    output: { x: 700, y: 10 }
  },
  // When close to bottom left corner
  {
    input: {
      clickX: 10,
      clickY: 720,
      viewportWidth: 1000,
      viewportHeight: 800,
      menuWidth: 200,
      menuHeight: 160
    },
    // move the context menu so that it is within viewport
    output: { x: 10, y: 560 }
  },
  // When close to bottom right corner
  {
    input: {
      clickX: 900,
      clickY: 720,
      viewportWidth: 1000,
      viewportHeight: 800,
      menuWidth: 200,
      menuHeight: 160
    },
    // move the context menu so that it is within viewport
    output: { x: 700, y: 560 }
  }
  // Add more data points for better prediction
];

const data = rawData.map(normalizeData);

neuralNetwork.train(data, {
  iterations: 20000,
  errorThresh: 0.005,
  log: (stats) => console.log(stats)
});

// It's preferred train neural network offline using node.js
console.log(JSON.stringify(neuralNetwork.toJSON())); 

// We will store the output of neuralNetwork.toJSON() into src/utils/trainedModel.json

Building the React Application

Create a new React application using Create React App or your preferred method. In the application, create a new component called ContextMenu that will display the context menu and handle its positioning based on the neural network predictions. The ContextMenu component should have its own state for the position and visibility of the context menu.

// src/ContextMenu.js

import React, { useState, useEffect, useRef } from "react";
import "./ContextMenu.css";

export const ContextMenu = ({ net }) => {
  const [menuPosition, setMenuPosition] = useState({ x: 0, y: 0 });
  const [visible, setVisible] = useState(false);
  const contextMenuRef = useRef();

  const handleClose = () => {
    setVisible(false);
  };

  useEffect(() => {
    const handleContextMenu = (event) => {
      event.preventDefault();
      const contextMenuBoundingRect = contextMenuRef.current.getBoundingClientRect();

      const input = {
        clickX: event.clientX / window.innerWidth,
        clickY: event.clientY / window.innerHeight,
        viewportWidth: 1,
        viewportHeight: 1,
        menuWidth: contextMenuBoundingRect.width / window.innerWidth,
        menuHeight: contextMenuBoundingRect.height / window.innerHeight
      };

      const prediction = net.run(input);

      const menuX = prediction.x * window.innerWidth;
      const menuY = prediction.y * window.innerHeight;

      setMenuPosition({ x: menuX, y: menuY });
      setVisible(true);
    };
    window.addEventListener("contextmenu", handleContextMenu);

    return () => {
      window.removeEventListener("contextmenu", handleContextMenu);
    };
  }, [net]);

  return (
    <div
      className="ContextMenu"
      onClick={handleClose}
      ref={contextMenuRef}
      style={{
        left: menuPosition.x,
        top: menuPosition.y,
        display: visible ? "block" : "none"
      }}
    >
      {/* Add your context menu items here */}
      <p>Menu Item 1</p>
      <p>Menu Item 2</p>
      <p>Menu Item 3</p>
    </div>
  );
};

export default ContextMenu;

Create a CSS file ContextMenu.css in the src folder for styling:

/* src/ContextMenu.css */

.ContextMenu {
  position: absolute;
  background-color: white;
  border: 1px solid #ccc;
  padding: 10px;
  box-shadow: 0 0 10px rgba(0, 0, 0, 0.1);
  z-index: 100;
}

Integrating the Model into the React Application

In the main App.js file, import the ContextMenu component and include it in the application's JSX. Pass the trained neural network instance as a prop to the ContextMenu component.

// src/App.js

import "./styles.css";
import ContextMenu from "./ContextMenu";
import { neuralNetwork } from "./utils/neuralNetwork";
import trainedModel from "./utils/trainedModel.json";

const trainedNetwork = neuralNetwork.fromJSON(trainedModel);

export default function App() {
  return (
    <div className="App">
      <h1>Right-click anywhere on the screen</h1>
      <ContextMenu net={trainedNetwork} />
    </div>
  );
}

Conclusion

In this article, we explored the possibility of using a neural network to predict context menu positions in a web application using the Brain.js library and React. While this approach is not recommended for practical use, it serves as a learning exercise to help you understand how to apply neural networks to various problems.

I hope that this post was interesting and informative. I would be glad to know how are you using machine learning and AI in your frontend applications. Please feel free to share your experiences and ideas in the comments section below.