findme 🕵️‍♀️🕵️‍♂️

findme is a serverless application to find unlabelled photos of you on twitter using machine learning.

Users provide a search query to retrieve tweets from the Twitter API. Face recognition is used to compare all faces found in the search results against the user's twitter profile image. Tweets with matching faces are shown in the client-side web application.

dependencies

If you want to deploy this project you will need an instance of the Apache OpenWhisk platform, access to a Redis database and credentials for Twitter and Auth0 applications.

Follow the instructions below to deploy this application on IBM Cloud (including dependent services). Auth0 and Twitter developer accounts need to be registered separately.

IBM Cloud

• Sign up for a free "Lite" account with IBM Cloud here.
• Install the IBM Cloud CLI

IBM Cloud Functions

IBM Cloud Functions is a managed instance of Apache OpenWhisk running in the public cloud. Lite account users have access to 400,000 GB/s of free compute time per month.

IBM Cloud Functions is available in the following regions: us-south, us-east, london and frankfurt.

• Log into the IBM Cloud CLI using the region endpoint chosen to deploy the application.

  ibmcloud login -a <REGION_ENDPOINTS>
  

• Install the Cloud Functions CLI plugin.

  ibmcloud plugin install cloud-functions
  

Redis on IBM Cloud

IBM Cloud provides managed Redis instances that charges solely for usage, i.e. no fixed monthly fee.

• Provision a new Redis instance through the IBM Cloud web console.

• Follow the documentation here to retrieve connection strings for the database.

Twitter App

• Use the Twitter Developer portal to register a new application.
• Retrieve the consumer_key and consumer_secret values for the application.

Auth0 Account

• Use the Auth0 application dashboard to register a new "Single Page Application".
• Retrieve the app_domain, client_id and client_secret values for the application.

installation

install project dependencies

• Install The Serverless Framework.

  npm install serverless
  

• Clone Git repository.

  git clone https://github.com/jthomas/findme.git
  

• Install project dependencies.

  cd findme && npm install
  

build custom runtime image (optional)

If you don't want to manually build a custom runtime image, you can use the following pre-existing image: jamesthomas/action-nodejs-v8:tfjs-faceapi. The serverless.yml is already configured to use this runtime image. If you need to make changes to the runtime image, follow these steps...

• Install Docker and sign up for an account at Docker Hub.

• Log into the public Docker Hub registry using the Docker client.

  docker login
  

• Build the custom runtime locally.

  docker build -t <DOCKERHUB_USERNAME>/tf-js .
  

• Export the local image to Docker Hub.

  docker push <DOCKERHUB_USERNAME>/tf-js
  

• Update the serverless.yaml image property with the new image name.

fill in authentication credentials

• Create authentication credentials for Redis, Auth0 and Twitter in creds.json file.

  {
    "redis": "redis://<REDIS_URL>",
    "twitter": {
      "consumer_key": "<CONSUMER_KEY>",
      "consumer_secret": "<CONSUMER_SECRET>"
    },
    "auth0": {
      "domain": "<USER_NAME>.auth0.com",
      "clientId": "<CLIENT_ID>",
      "clientSecret": "<CLIENT_SECRET>"
    }
  }

deploy backend

• Run the deploy command.

  serverless deploy
  

• Retrieve API Gateway endpoint (https://<APIGW_URL>/findme) from deployment logs.

  endpoints (api-gw):
  GET https://<APIGW_URL>/findme/api/search/{id} --> search_status
  POST https://<APIGW_URL>/findme/api/search --> schedule_search
  

configure client-side app

• Update CONFIG value in public/script.js with API Gateway URL and Auth0 application identifiers.

  const CONFIG = {
    auth0: {
      clientId: '<CLIENT_ID>',
      domain: '<USER_ID>.auth0.com'
    },
    backend: 'https://<APIGW_URL>/findme'
  }

run web server for client-side app

• Start web server to host static files in public directory, e.g.

  python -m SimpleHTTPServer
  

• Open index.html on web server. 👍

architecture

This application has four serverless functions (two API handlers and two backend services) and a client-side application from a static web page.

Users log into the client-side web page using Auth0 and a valid Twitter account. This provides the backend application with the twitter profile image and API credentials.

When the user invokes a search query, the client-side application invokes the API endpoint for the register_search function with the query terms and twitter credentials. This function registers a new search job in Redis and fires a search_request trigger with the query and job id. This job identifier is returned to the client to poll for real-time status updates.

The twitter_search function is connected to the search_request trigger and invoked for each event. When this function is invoked, it uses the Twitter Search API to retrieve all tweets for the search terms. If the tweets contains photos, each tweet and photo url is fired as a separate tweet_image trigger event.

The compare_images function is connected to the search_request trigger. When this function is invoked, it downloads the user's twitter profile image along with the tweet image and runs face extraction against both images. If any faces in the tweet image match the face from the user's profile image, tweet ids are written to Redis before exiting.

The client-side web page polls for real-time search results by calling the API endpoint for the search_status function with the job identifier for the search. If tweet ids are returned from the search results, those tweets are displayed on the web page using the Twitter JS library.

limits

• Twitter Standard Search API only provides access to tweets from the last seven days. Access to older tweets is only available by upgrading to premium or enterprise tiers.
• The Twitter profile image for the authenticated user is used as the sample face to match against photos in search results. If your profile image is not your face, it will not find you...
• Search results for a query will retrieve at most 1000 tweets. This limit can be changed in the lib/twitter/api.js file.

issues / feedback / feature requests?

If you have any issues, comments or want to see new features, please file an issue in the project repository:

https://github.com/jthomas/findme