A demo application that can aggregate search results from Google, DuckDuckGo and Wikipedia for a given search keyword
Design diagram
Search Page
Search Listing
Search Pending
Search Results1
Search Results2
Note I have tested this way on a Mac Book Pro with 16Gb RAM.
There are total 9 micro services as listed below
- coolsearch-api - golang
- ddg-search-service - Python
- google-search-service - Python
- wiki-search-service - Python
- search-aggregator - golang
- search-persistance - golang
- coolsearch-result-api - golang
- coolsearch-result-notifier - Nodejs
- coolsearch-app-server - JS, Nodejs, React
The following dependencies should be resolved before running the start.sh script
librdkafka - This is a dependency for confluentinc/confluent-kafka-go.v1/kafka
Run the following to resolve this dependency on host machine. (Should run on Mac and linux)
Run this only on Ubuntu apk -U add ca-certificates apk update && apk upgrade && apk add git bash build-base sudo
Linux/Mac git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && ./configure --prefix /usr && make -j && make install
BeautifulSoup4 - This is a python dependency
Run pip/pip3 install beautifulsoup4
To fix Nodejs dependencies for services 8,9 run npm install
Golang Dependencies - To resolve these need to run 'go get' on below.
go get github.com/go-chi/chi go get gopkg.in/confluentinc/confluent-kafka-go.v1/kafka go get github.com/go-redis/redis go get go.mongodb.org/mongo-driver/mongo go get gopkg.in/mgo.v2/bson go get github.com/go-chi/cors
First startup the infrastructure containers using docker compose by running below command
docker-compose up
Running the Micro services - (Run the above steps first to resolve dependencies)
Run the nodejs services by running 'npm start' Run the Python services by running 'python3 ' (Python 3 is required) Run the Golang services by running 'go run '
Create a startup.sh & shutdown.sh file that can startup and shutdown the services. Complete Dockerization and create docker-compose.yml to run all services together.