Search - discussion 🐓

[andrmdmd]

Here is a space to talk about the search functionality.

For now, planned fields are:

Filtering:

• provider
• brand
• volume (from, to)
• rating (from, to) (?)
• ingreedients (to include, to exclude)

Sorting:

• liked first
• highest rating first (?)

Feel free to give your opinions about these!

[mslup]

I would also like to clarify the distinction between searching and filtering. As I understand it right now, searching should be done in the following way -- the user inputs a set of space-separated keywords, each of which is searched in:

• name,
• short description,
• long description,
• brand,

and all the products that contained the keyword in one of the above are returned.

Does the above mean that filtering by brands is obsolete?

Should we filter by a list of providers (of predefined ones) instead of individual ones?

[andrmdmd]

As for the brand, I mean filtering by brand = searching the keywords there. There is no need for a separate field for a brand (either on frontend or in query params sent to backend)

[mslup]

Maybe we should implement Polish declension 🤠

[andrmdmd]

Generally I think searching in the long description may introduce performance issues, as some descriptions are quite long (eg see yapping in https://www.rossmann.pl/Produkt/Serum/Dermika-Luxury-Ceramides-serum-do-twarzy-ceramidowe-przeciwzmarszczkowe-w-kapsulkach-dzien-noc-,409858,9225). I do not know how the search is performed on backend, but when we have data imported, it should be something we check for performance.

[migoox]

Does the above mean that filtering by brands is obsolete?

I think that we can have both -- filtering by brand (we will need the sql table) AND searching which takes the brand into account. The user should be able to choose if they want to do quick search which involves writing something like pentene shampoo in the search bar, or to do filtering which may provide higher quality of the results. It feels more user friendly to me

[kubazuch]

I agree with @migoox, filtering and searching by brand are not mutually exclusive, rather they complement each other.

[mslup]

Mentioned searching here: #28 (comment)

[andrmdmd]

Another thing we should discuss about filtering - volume. Our data has varied typed of products, so we can expect units like:

• ml,
• g,
• szt,
• kg,
• unconventional formats, e.g. 3x500ml.

This seems to be hard to standardize. We could think about something like small, medium, large for every format, but I do not believe it would yield good results (e.g. large for nail polish vs large for shampoo for ml unit).

[mslup]

Is there a user story for filtering by volume? If so, perhaps treating it like we treat other string fields would be the straightforward solution.

[andrmdmd]

Do we have user stories for anything?

[migoox]

I think that this user story is irrelevant in our application, we are focusing on different aspects of products than price, volume etc. We can provide this info on the product pages if it was obtained easy, but I think that the filtering by this info should be "won't have"

[migoox]

Endpoint specification proposal

This specification doesn't take the technology we are currently using into account. It means that this is ideal plan of our search, the goal we are trying to converge to -- not the final specification of the endpoint.

Endpoint signature

Signature:

GET /api/products?phrase=<phrase-string>&providers=<comma-separated-indices>&brands=<comma-separated-indices>&min-rating=<rating-from-num>&ingredients-include=<comma-separated-indices>&ingredients-exclude=<comma-separated-indices>&categories=<comma-separated-indices>&sort-by=<comma-separated-sorting-enums>

Where all params are optional.

Example:

GET /api/products/?phrase=skin%20care&ingredients-include=60,30&min-rating=8&categories=1,2&sort-by=d-match-score

Params description:

1. Filtering:
   • ingredients-exclude (must have): string that contains comma separated indices of ingreedeints in our sql db, for example 55,33,12,56,2,
   • ingredients-include (must have): string that contains comma separated indices of ingreedeints in our sql db,
   • categories (should have): string that contains comma separated indices of categories in our sql db,
   • min-rating (should have): 1-10 number indicating minimum rating of a product,
   • provider (could have): string that contains comma separated indices of providers in our sql db,
   • brand (could have): string that contains comma separated indices of brands in our sql db.
2. Searching:
   • phrase (must have): contains searching phrase string, every space is replaced with %20,
3. Sorting:
   • sort-by (should have): string that contains comma separated enum variables indicating the sorting options, the order in which the variables are provided indicates the sorting order. More in Sorting mechanism section.

Note: The order of the parameter types is not accidental; the endpoint will first process filtering, then searching, and finally sorting.

Filtering mechanism

Filtering is pretty obvious and self-descriptive -- it allows excluding products that doesn't meet the criteria from the returned products set. The resultant set is then passed ot the searching stage.

Searching mechanism

If phrase param is provided in the endpoint url, then the backend will search for products. Searching consists of two stages:

1. finding matching products,
2. finding the match-score, which is described later.

The products are first filtered -- only those that are matching are in the resultant set. For each matching product the match-score is found and is passed to the sorting stage.

Note: The match-score is equal 0 for every product if the phrase is not provided (is null or empty).

Finding matching products

Product is said to be matching if it meets the following rule:

at least one word in the phrase matches any word in name, brand or short description fields.

Note: The word matching isn't case sensitive, for example WoRD matches woRd.

Finding match score for each product

The match-score is defined basing on the the following rules:

1. For every unique occurence of the word inside
   • the brand field, match-score += 20
   • the name field, match-score += 15
   • the short description field, match-score += 5
2. Any ideas?

The match score can be used in the sorting to get better results.

Sorting mechanism

The sorting mechanism allows sorting by the following enum variables:

• a-rating, d-rating: sort by the rating of the product,
• a-rate-count, d-rate-count: sort by the number of the ratings,
• a-opinions-count, d-opinions-count: sort by the number of the opinions,
• a-match-score, d-match-score: sort by the match score, found during the searching. If phrase is null or empty the match-score equals 0 for every product, so the sorting won't cause any changes,

where a indicates ascending order and d indicates descending order.

More than one variable may be provided -- the order of these variables matters:

For example, if the query is a-rating,d-opinions-count, then products will be sorted in ascending order of the rating. If there are products with the same rating, they will then be sorted in descending order by the opinions-count.

[GitQueenZofia]

Great job @migoox ! It seems really good for me. But i think filtering by brand should be "should have" because I personally use it very often but that's just my opinion. Also when we find match score for each product I think name is as important as brand (or even more important) when we search for a specific product but I would like to hear what others think. But things I pointed out are only details, I'm really proud of your job!

[migoox]

Thank you for participating in that discussion and for your feedback @GitQueenZofia!

1. I've decided to use "could have" for filtering by brand as it produces additional work (both frontend and backend + extracting data from dataset), while not being as beneficial, since users may use searching in order to find products by their brands. I believe in iterative process, which means that we are not abandoning the idea (it's good since it allows user to do quality searches), we are doing the most important things first.
2. I think that we can figure out what are the best match score rules by practice, I will try to do the implementation configurable to make tweaking the values easier 😄

[kubazuch]

Amazing work, I don't have anything to add.