VF

E-commerce Search

Jul 11, 2024

Fast search solution with OpenSearch and Graph database Neptune for the e-commerce platform.

Search Image Search Design Schema

In today’s competitive e-commerce reality, customers demand fast, accurate, and personalized search experiences. To address these challenges, our project leveraged AWS OpenSearch and Neptune to create a scalable, high-performance search solution tailored for e-commerce platforms. This implementation not only improves search speed and relevance but also empowers businesses with advanced filtering, categorization, and enrichment capabilities.

The business need

As e-commerce platforms grow, the volume and complexity of their data increase exponentially. Customers often search for products, categories, or offers using incomplete or vague keywords. To meet these demands, a robust search engine must deliver results quickly, accurately, and with meaningful context. Our solution addresses these needs, ensuring seamless navigation for users while enabling administrators to efficiently manage categories, filters, and enrichments.

Solution design

The implemented architecture combines the indexing power of AWS OpenSearch with the relationship-based querying of AWS Neptune. This dual-stack approach allows for fast, full-text search and complex graph-based operations, such as category enrichment and personalized recommendations. The solution integrates seamlessly with the e-commerce platform, providing tailored results and reducing friction in the customer journey.

Core features

  • Advanced Search Capabilities: Full-text search with support for multi-level filtering, sorting, and pagination to deliver precise results for customers and administrators alike.
  • Real-Time Suggestions: Auto-suggestion for products and categories as users type, improving search accuracy and reducing effort.
  • Graph-Powered Enrichment: Using Neptune’s graph database, categories and products are enriched with detailed breadcrumbs, relationships, and metadata, enhancing the user experience.
  • Admin Control: Robust tools for administrators to manage categories, define filters, and oversee product sorting mechanisms.
  • Scalable and Cost-Efficient: OpenSearch and Neptune are configured to scale dynamically, ensuring cost-effectiveness without compromising performance.

Technical implementation

The solution is built on AWS cloud services, ensuring high availability, security, and scalability. Here’s a high-level overview of the technical components:

  • AWS OpenSearch: Configured for full-text indexing, filtering, and sorting with custom access policies for secure integration with Lambda functions.
  • AWS Neptune: Deployed with subnet groups and security configurations to support graph-based queries for category enrichment and relationships.
  • Serverless Architecture: Leveraging AWS Lambda for event-driven interactions between OpenSearch, Neptune, and the e-commerce application.
  • Dynamic Scaling: OpenSearch instance types and Neptune clusters are optimized for resource usage, ensuring performance under varying loads.
  • Flexible API Integration: Comprehensive endpoints for administrators and customers, supporting query-based interactions and data enrichment.

Example templates

AWS OpenSearch template

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Parameters:
  OpenSearchDomainName:
    Type: String
    Default: search-service
  OpenSearchDomainEndpoint:
    Type: String
    Default: http://host.docker.internal:4566/opensearch/search-service
  OpenSearchInstanceType:
    Type: String
    Default: t3.small.search
    AllowedValues:
      - t3.small.search
      - t3.medium.search
    Description: Enter t2.micro, m1.small, or m1.large. Default is t3.small.search.
  OpenSearchEbsVolumeSize:
    Type: Number
    Default: 10
    AllowedValues:
      - 10
      - 20
    Description: The size (in GiB) of the EBS volume for each data node

Resources:
  OpenSearchDomain:
    Type: AWS::OpenSearchService::Domain
    Properties:
      DomainName: !Ref OpenSearchDomainName
      ClusterConfig:
        InstanceType: !Ref OpenSearchInstanceType
        InstanceCount: 1
        DedicatedMasterEnabled: false
        ZoneAwarenessEnabled: false
      AccessPolicies:
        Version: '2012-10-17'
        Statement:
          - Effect: 'Allow'
            Principal:
              AWS: "*"
            Action: "es:ESHttp*"
            Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${OpenSearchDomainName}/*
      EBSOptions:
        EBSEnabled: true
        VolumeType: gp2
        VolumeSize: OpenSearchEbsVolumeSize
      DomainEndpointOptions:
        CustomEndpoint: !Ref OpenSearchDomainEndpoint
        EnforceHTTPS: false
        CustomEndpointEnabled: true

  LambdaOpenSearchPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action: "es:ESHttp*"
            Resource: !GetAtt OpenSearchDomain.Arn

Outputs:
  OpenSearchDomainName:
    Value: !Sub ${OpenSearchDomainName}
  OpenSearchEndpoint:
    Value: !Sub https://${OpenSearchDomain.DomainEndpoint}

AWS Neptune template

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Parameters:
  ServiceName:
    Description: 'Service Name (default neptune)'
    Default: 'neptune'
    Type: String
  VpcId:
    Description: 'Vpc Id'
    Type: String
  SubnetIds:
    Description: 'Subnet Ids'
    Type: String
  NeptuneInstanceClass:
    Type: String
    Default: db.t3.medium
  NeptunePort:
    Default: 8182
    Type: Number

Resources:
  NeptuneSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    DependsOn: LambdaSecurityGroup
    Properties:
      GroupName: "neptune-security-group"
      GroupDescription: Allow access to Amazon Neptune from Lambda
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 0
          ToPort: 65535
          SourceSecurityGroupId: !Ref LambdaSecurityGroup
      VpcId: !Ref VpcId
  LambdaSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: "lambda-security-group"
      GroupDescription: Access from AWS Lambda to Amazon Neptune
      SecurityGroupEgress:
        - Description: Egress port of the https
          IpProtocol: tcp
          CidrIp: 0.0.0.0/0
          FromPort: 443
          ToPort: 443
        - IpProtocol: tcp
          FromPort: 8182
          ToPort: 8182
          CidrIp: 0.0.0.0/0
      VpcId: !Ref VpcId
  NeptuneSubnets:
    Type: AWS::Neptune::DBSubnetGroup
    Condition: NonLocalCondition
    Properties:
      DBSubnetGroupDescription: "Neptune Subnet Group"
      DBSubnetGroupName: "neptune-subnet-group"
      SubnetIds: !Split [',', !Ref SubnetIds]
  NeptuneCluster:
    Type: AWS::Neptune::DBCluster
    Condition: NonLocalCondition
    Properties:
      DBClusterIdentifier: 'neptune-cluster'
  DBSubnetGroupName: !Ref NeptuneSubnets
      AssociatedRoles:
        - RoleArn: !GetAtt NeptuneLoadFromS3Role.Arn
      VpcSecurityGroupIds:
        - !Ref NeptuneSecurityGroup
  NeptuneInstance1:
    Type: AWS::Neptune::DBInstance
    Condition: NonLocalCondition
    Properties:
      DBClusterIdentifier: !Ref NeptuneCluster
      DBInstanceClass: !Ref NeptuneInstanceClass
      DBInstanceIdentifier: !Sub "neptune-instance"
      DBSubnetGroupName: !Ref NeptuneSubnets
  NeptuneLoadFromS3Role:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: !Sub "neptune-load-from-s3-role"
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - rds.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: /
  NeptuneLoadFromS3Policy:
    Type: 'AWS::IAM::Policy'
    Properties:
      PolicyName: !Sub "neptune-load-from-s3-policy"
      PolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - 's3:Get*'
              - 's3:List*'
            Resource: '*'
      Roles:
        - !Ref NeptuneLoadFromS3Role
  NeptuneS3VpcEndpoint:
    Type: 'AWS::EC2::VPCEndpoint'
    Properties:
      VpcEndpointType: 'Gateway'
      VpcId: !Ref VpcId
      ServiceName: !Sub 'com.amazonaws.${AWS::Region}.s3'
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal: '*'
            Action:
              - 's3:GetObject'
            Resource: '*'
Outputs:
  NeptuneEndpoint:
    Value: !If [NonLocalCondition, !GetAtt NeptuneCluster.Endpoint, "neptune"]
  NeptuneReadEndpoint:
    Value: !If [NonLocalCondition, !GetAtt NeptuneCluster.ReadEndpoint, "neptune"]
  NeptunePort:
    Value: !If [NonLocalCondition, !Ref NeptunePort, 8182]
  NeptuneProtocol:
    Value: 'wss'
  LambdaSecurityGroup:
    Value: !If [NonLocalCondition, !Ref LambdaSecurityGroup, '-']

Summary

This search engine implementation showcases how modern cloud technologies can transform e-commerce platforms. By combining OpenSearch’s indexing capabilities with Neptune’s relationship-based querying, the solution delivers fast, intuitive, and scalable search experiences. Whether you’re a customer looking for the perfect product or an administrator managing a complex catalog, this architecture ensures efficiency, relevance, and ease of use at every step.