Search functionality wishlist

[rdn]

Current search is very limited, i want to find out what requirements i need to work with.

[ul]
[li]At the present moment, v5 will use Elasticsearch[/li]
[li]There are 5 endpoints that will support search queries, /search/manga, /search/chapter, /search/author, /search/user, /search/group[/li]
[li]I will not directly pass through search queries to elasticsearch because i dont know if this can be abused. Let me know if you have contrary information[/li]
[li]resource ids will change from int to uuid[/li]
[/ul]

I've identified a set of use cases that i think will cover most of the search needs:

POST /search/manga
- query: "*" (default) | any non-empty string,
- sort:
--- relevancy
--- created
--- updated.<languageCode>
--- title.<languageCode>
--- year
--- chapterCount.<languageCode>
- filter:
--- isLocked: bool
--- author: uuid
--- artist: uuid
--- originalLanguage: string (ex: "jp", "en") 
--- hasEnded: bool
--- publicationDemographic: string
--- status: string (ex: "ongoing", "hiatus")
--- year: range
--- contentRating: string (ex: "hentai")

POST /search/chapter
- query: "*" (default) | any non-empty string,
- sort:
--- relevancy
--- created
--- vol/chapter number (default)
- filter:
--- language: string, ex: ("en", "ru", "it", ...)
--- uploader: uuid
--- groups: uuid
--- manga: uuid

group, user and artist are undetermined.

For the endpoints i mentioned, do you think this works? Are you missing any particular query that isnt covered?
Would you make something different?

Let me know

 

[Tamerlane]

I think a lot of those are needed, but we also should amend the tags to include other groups.

Something like user-submitted tags like steam has that mods can pick from the most common of or expanding the options we have to be more specific could be needed.

For instance, I've noticed the tag "supernatural" is very overly inclusive and not specific enough, and so it covers a lot of manga with very little in common. I'd recommend supplementing it with something similar to dividing the "fantasy" tag into "high fantasy," "low fantasy," "Urban Fantasy," etc. to get a more specific idea. Alternatively, having sub-tags that divide into more specific groups could help, such as instead of just "supernatural," that may describe one mundane power someone might have versus a complex magic system within the setting may include sub-tags such as "down-to-earth," "mystical," etc. that could further specify. Drama is another one, because it could include anything from a modern romcom with relatively low stakes to a Shakespearean tragedy or a detailing of the fallout of society into a political system that could lead to the rise of an oppressive regime. The fact most tags are overly inclusive is fine for generalist searches into a genre, but subsearches for more specificity could be more useful and ergonomic.

I'm spitballing but I desperately thing reworking how tags function could be a good idea, especially if we go for specific archetypes in manga like "tomboy" or "bara" which is its own can of worms.

 

[EOTFOFYL]

--- contentRating: string (ex: "hentai")

Out of curiosity, what are the possible values for content rating? There's probably more than two otherwise I'd assume it would have been a bool.

 

[Tamerlane]

@EOTFOFYL

I'd imagine:
-Ecchi
-Gore
-Sexual Violence
-Smut

+ Other ones

 

[JJTheWeeb]

@EOTFOFYL it was mentioned in other pages, but it's a ecchi bar, so like:
tame(rlame) to hentai, kinda like this:
nothing (rated everyone under 8) - kissing - handholding - ecchi - borderline - porn - good god why
or smt. maybe it changed, but yea
i did use a little memespeak, but whatevs

 

[rdn]

@Tamerlane good call with the tags. we have them planned but i kinda forgot about them because they arent modeled in our databases yet. I'll try to update the list in OP soon.

Generally I dont see a reason to overengineer them in terms of how they interact with the search. There might be a way where hierarchies come into play, lets say you have tag A and Subtags b,c,d. When you search for tag A it could include b,c,d. But that's only a shortcut and not worth the trouble imo.
At the end of the day, you want to search for manga that have the tags you specified. if you filter for [A, B] then it only matches manga that have both A and B. Subtagging would otherwise serve a pure visual function which I'm not really concerned about right now.
One other issue is how to do globally excluded tags, but those will just have to be put into each search query to keep its stateless nature.

@EOTFOFYL
i think it was related to the kinds of ratings Tamerlane has listet. Not exhaustively, i'm sure we'll get more together. @Teasday can say more to it, i think the attribute was from him

 

[Lexical]

can Elasticsearch work with the 'End' tag for series that have uploaded chapters tagged as such?

 

[Teasday]

@EOTFOFYL We had the big brain idea of calling it the SEX rating: Safe, Erotica and Explicit. There might have been a fourth tier as well, can't remember right now. Either way, whatever the terms and however many tiers we'll use will be defined eventually.

We don't want to use vague terms like ecchi and hentai because people seem to have very strong opinions about what counts as them in order to argue that certain very explicit material isn't actually hentai so it shouldn't be marked as such.

 

[rdn]

@Lexical can you make an example what you would expect in relation to the end tag? how would you want to use the search where it is relevant?

 

[Lexical]

We already have a series status “Completed” tag but that’s currently used by most series on MD to denote the publishing status ie ongoing, canceled, hiatus, or completed. As is, there are many series that are marked completed but are inactive projects but since they all were uploaded 3 years ago their ‘last updated’ date doesn’t give any useful information either

If a chapter’s ‘End’ tag status can be indexed upon receiving the tag then the search () could return only series that have been completely scanlated (or at the very least least those which have had the final chapter uploaded to the md database and flagged as such )

 GET /_search
{
  "query": {
    "simple_query_string" : {
        "query": "\"Magic school bus "\"  -mulan",
        "fields": ["title^5", "body"],
         "tags": ["LanguageIsKlingon","ChapterEndTagKlingonTrue ", "IsekaiIsTrue"],
         "default_operator": "and"
    }

.sorry, it’s been years since i’ve coded anything so I’m not too clear how ridiculous or simple such a task may be nowadays but something like that..


Also possible I misunderstood because my lexicon is so old and

 --- hasEnded: bool

<—... and this line is actually the same as what i was thinking of

--- publicationDemographic: string
--- status: string (ex: "ongoing", "hiatus")

<— and this has “cancelled”, “completed”

 

[bigtiddyoneesan]

There might have been a fourth tier as well, can't remember right now

SEXY rating:
Safe
Erotica
Explicit
You've been warned (for guro and shit)

 

[Teasday]

_{@Lexical posted:}

If a chapter’s ‘End’ tag status can be indexed upon receiving the tag then the search () could return only series that have been completely scanlated (or at the very least least those which have had the final chapter uploaded to the md database and flagged as such )

While true, I would be reluctant to add it out of fears of it emboldening people to snipe the last chapters for trolling purposes when the scanlation is way behind. It already happens, making it a site feature to find manga with a scanlated final chapter sounds to me like an invitation to make the problem worse.

 

[Halo]

_{@Teasday proclaimed:}

...snipe the last chapters for trolling purposes

Then this is okay, because troll releases are expressly forbidden by the rules:

2.3.5: Joke or troll releases are not allowed, except on April 1st.

I can't believe we're finally getting this long requested feature.

 

[Teasday]

@Halo Separate matter, but you already knew that

 

[Halo]

@Teasday If there's a type of behavior that prevents you from implementing a feature, why not just make a rule against it?

 

[EOTFOFYL]

@Halo

Rules without power is meaningless and even then, will not stop people from trying.

 

[Halo]

@EOTFOFYL They have power: moderation. Troll releases are forbidden and they're being removed just fine. Besides, not having this feature doesn't stop feral assholes either.
I don't even care about the search feature tbh, it's just annoying to see last chapters getting sniped.

 

[Lexical]

I concede that there have been instances of abuse, but on the other hand, and perhaps I’ve just been fortunate, but I’ve only encountered much less than a dozen such trolls chapters since mangadex started and most of those were mostly during 2018 if my memory hasn’t failed .

Granted, since I’m just a user and don’t have to deal with “mod mail” i’ve no idea how many such reports there are per month

 

[Teasday]

_{@Halo posted:}

If there's a type of behavior that prevents you from implementing a feature, why not just make a rule against it?

We don't want to. The intent may be trolling, but if the chapters are legitimate scanlations, we don't really have a reason to deny them. Not our place to tell people what chapters they are and aren't allowed to scanlate.

 

[Halo]

@Teasday Yet, you're trying to discourage this behavior by intentionally hindering the searching capabilities of this website.