FANDOM


  • Hello. I've been working with Ar-cen-ciel for a long time at the FKG Wiki. We've suffered from problems of wrong information popping up in pages due to copy-pasting of data here and there. Currently, a bunch of scripts Ar-cen-ciel uses would generate a big Template call for character pages, then editors/translators would put changes on top of that. (Example: Curcuma's character page). Pages involving data based on those characters, like a List of Shared Abilities, would be done via copy-paste.

    I wanted to alleviate those problems by having a database of some sort for all of the game's characters. The solution I had thought of was to store it all in a couple Lua tables similar to the work in progress that is Module:Character. Before jumping into too much stuff, I thought I'd look at how this Wikia handles its character info. There's some questions I have regarding all this. Some of the questions may have overlapping answers.

    1. Does KanColle's Wikia store its character statistics in Lua tables?

    2. I tried following a character's page to see how the infoboxes were made (Specifically Inazuma). The flow that I could gather was...

    (Inazuma's page)
    {{ShipInfoKai|Inazuma/|los=auto|los_max=auto|evasion=auto|evasion_max=auto|asw=auto|asw_max=auto}}
    
    ...
    (Template:ShipInfoKai)
    {{ {{{subst|safesubst:}}}#invoke:ShipInfo|Infobox}}
    
    ...
    (Module:ShipInfo)
    function ShipInfo.Infobox(frame)
    	local args = getArgs{frame = frame:getParent()}
    	_prepareShipInfoKai()
    	return ShipInfoKai:Infobox(args)
    end
    

    I believe I understand _prepareShipInfoKai(), but not the two lines surrounding it. I'm guessing frame:getParent() jumps up from ShipInfoKaika to Inazuma's page, so it gets the stuff sent to ShipInfoKai, but how does ShipInfoKai:Infobox(args) work? Specifically, I was curious how it would know Inazuma's parameters for the infobox.

    3. A big concern of Ar-cen-ciel and mine is that Lua tables don't seem as user-friendly for editors to update as the Template we use now. Is there a solution or workaround to that? There's some user-inputted data we can get away with only showing on Character pages and nowhere else (like skill name translations), but some information shows up on multiple pages (like English translations of the flower knight's name; there can be multiple).

    4. Are these Templates that safe-substitute a Module invocation what you call a proxy module? What is the purpose of that? Also, if you change any of the Modules, wouldn't that leave the Template-calling page unchanged because of the substitution?

      Loading editor
    • 1. Yes, with a module/Lua table per ship (Category:Player ship modules, e.g. Module:Inazuma, same for equipment and some other things).

      2. Module:ShipInfo requires an utility module Module:GetArgs, so the getArgs line gets the arguments from the initial call (i.e., ShipInfoKai, not #invoke:ShipInfo, there is getTemplateArgs function in Module:Utils that can get both, which is useful when you want to pass a page name implicitly, e.g., to have {{ShipInfoKai}} instead of {{ShipInfoKai|Inazuma}}), mainly [1], which is Inazuma/ in this case. _prepareShipInfoKai also requires Module:ShipInfoKai, which has ShipInfoKai:Infobox defined in it and also depends on other modules. You probably can implement something similar in a simpler way (and then complicate it as needed, what we have is KC specific in many places), for example, in Module:Utils there is requireModule function, so you can do local exists, data = Utils.requireModule("Inazuma") and use data (if exists) in your formatter (we do have an abstraction layer though, the data is accessed via Ship (also has some caching) object's accessor functions defined in Module:ShipData, it can be useful, since that layer can be changed when needed without touching all the data modules).

      3. Well, what we have is edit buttons that take you to Lua tables, ability to override Lua data from template calls (those los=auto, etc.), then there is an argument that this stuff should be generated from API, so you don't need user edits in the first place, finally (IMO), wikitext is also a language and can be somewhat complicated if there is a Lua parser behind it, so it is not very different for a user whether it is a bunch of stuff in wikitext or in a Lua table (can break both).

      4. The purpose of templates like ShipInfoKai is to have a nicer interface (using ShipInfoKai is nicer than doing invoke everywhere, also safesubst doesn't substitute when transcluded, but provides viewing for templates and recursive substitution when actually substituted) and also to provide documentation (Template:ShipInfoKai/doc, included via Template:Documentation). What I call proxy modules is different (and they aren't really used widely at the moment), normal data modules store data that is pulled from different places, now say you want to store data in pages (not modules) and pull it from other pages, in simple cases you can use tags like includeonly and parser functions like if, then there is DPL (though I'm not sure about using it extensively on Wikia), and then you can keep generating associated "proxy" modules for your initial pages, so you kinda pull the data from those initial pages via those modules (good if you have one or few initial pages and many client pages, so you only generate those few proxy modules and not bot-edit many client pages).

        Loading editor
    • 2. To be exact, getParent gets you to Inazuma page, yes, and getArgs returns parsed arguments for that frame. Using {{{{{subst|safesubst:}}}#invoke:ShipInfo|pagename={{PAGENAME}}|Infobox}} with Utils.getTemplateArgs and {{ShipInfoKai|foo=bar}} call will give args.implicit.pagename == "Inazuma" (if on Inazuma page) and args.exlicit.foo == "bar".

        Loading editor
    • Thank you for the detailed response. I have a better understanding of how this works here and there, although I will need to study it more closely later. All the indirection is tough to swallow. For now, I'll definitely start with something simpler.

      The Lua tables being in each character's own module is good news to me because I'd like to do the same thing rather than having all character data in a massive list. I see how it's done here with the [Edit] link on the Infobox pointing to the module. There's some concerns I have though.

      1. FKG has ~340 characters and this grows by 4 characters minimum every other week. It'd be easy to add all 4 at once to a big table, but it would be awful for editors to update. By contrast, if I chose to make a data module for each character, I'd have to make over 340 modules for existing characters and an extra 4 every half-month... Is it possible to automate this? For the record, I can imagine the flow of translating the Template data already on those pages programmatically: Scrape the List of Characters, download individual pages using &action=raw -> translate to a Lua table -> upload to Wikia on the individual's module page. It's the last step that I have no idea how to do.
      2. If I had a module for every character, I worry about how much of an impact that would have if I loaded every module for every character. I can make a list of all character IDs and then require each of their modules. Is requiring (or at least building the lists for each of them) an expensive operation or should I not worry about that?
        Loading editor
    • We have around 600 various data modules, including around 200 player ship modules and 200 player equipment modules. On pages like Ship List (Image) ship modules are required over 400 times (because there can be different ship forms in the same module, on Equipment List (Image) it's just over 200 times, same as the number of equipment modules, the designators for creating Ship and Equipment objects from data modules are <English base ship name or form alias>[/[<ship form suffix>]] and <English equipment name>), the current Lua statistic for that page is

      NewPP limit report
      Preprocessor node count: 689/300000
      Post‐expand include size: 70598/2097152 bytes
      Template argument size: 0/2097152 bytes
      Expensive parser function count: 0/100
      Lua time usage: 0.300s
      Lua virtual size: 15.63 MB / 350 MB
      Lua estimated memory usage: 0 B
      

      Generally, many calls to Lua is bad on Wikia, while one or few calls is ok, even if heavy.

      For automation, I use modified version of https://github.com/macbre/nodemw (for creating new pages there is edit function, which is a wrapper for action=edit, it should be called in logIn callback if you want it to edit as a specific user, e.g., to have noratelimit right), but any mediawiki API library should suffice.

        Loading editor
    • One question: How does Kancolle Wikia get by with modules for each and every characters? CodeHK had been in favor for a single module until yesterday.

        Loading editor
    • I don't know how Ckwng created the first batch of data modules, but now we create and maintain a module per each entity (character, equipment, etc.) on each game update. What can be automated here:

      • API to module converters (what is not in API is edited manually afterwards), you don't actually write new modules.
      • API vs. module checkers, can check if any (API) stats have changed that way.
      • More complicated finders and checkers for data. E.g, in case of KC, finder/checker for soft stats (ASW/evasion/LoS) can be a thing, it should collect and work with certain data, not just convert an API object from a master file. Another example is Poi DB.
      • Operations on asset files associated with data modules (uploading, renaming, deleting, redirecting).
        Loading editor
    • What I'm asking is how do we get multiple modules to work without performance penalty caused by looking up hundreds of modules at once?

        Loading editor
    • Hm, I don't know, why there should be performance penalties?

        Loading editor
    • Using Ship List (Image) as an example, it works how it works, time and memory happen to be 5-10% of the limit (loosely for time, if we assume 3s is already too much, also assuming that both time and memory grow linearly, maybe they don't and it's even better), some arbitrary numbers.

        Loading editor
    • がか wrote: Hm, I don't know, why there should be performance penalties?
      Supposedly each module invocation would cause a performance hit that would make the pages that references multiple modules become slower. Seems like such a worry was pretty much unwarranted.
        Loading editor
    • Here is a graph.

      • Blue line is when you #invoke Lua over and over for the same data module (unrealistic, can have over 3000 invokes).
      • Red line is when you #invoke Lua for different data modules (maximum 200 in this test, then re-invoke for those 200 modules, it's dead at around 400-500 anyway).
      • A mix (same or different modules) is in the region in between.

      You almost never do this though. Let's call #invoke external call, then you usually do few to ~10 external calls to Lua, all other stuff is internal, i.e., via require, so, for example, for enemy patterns you do invoke 3 times (for easy, medium, and hard) and then require data from data modules up to 1000 times internally (for different enemies, maybe including duplicates, cached or not). Green dots are for proper templates that do it like that, e.g. (some arbitrary examples for existing pages)

      • Stuff like ship pages, equipment pages, etc., also drop lists and other small lists (like CV), one to few external and few to ~50 internal queries, < 5% for time and memory (using 3 seconds for 100%).
      • Big lists, one to few external queries, ~500 internal queries, ~10% time, still ~5% memory.
        • Can split it in ~20 external queries with 20-30 internal queries each, will be more time (like 1 second) and memory (~10%), but still ok, though it's always better to reduce the number of invokes.
      • Enemy patterns example is 2 seconds and 50/350 ~ 15% memory.
        Loading editor
    • I don't know if Ar-cen-ciel missed my original point or is just using the wrong terminology, but the thing I was talking about with him was the potential performance difference between a single require versus countless requires. The plan was already to have just a single invoke on each of the pages we wanted to automatize.

      Normally I would avoice file I/O (including things such as includes/requires) as much as possible becasuse disk access is the slowest type of memory and the most whimsical in time consumption, but I think that's because I was taught by programmers from an era where that was a big issue and it rubbed off on me.

        Loading editor
    • I mentioned many invokes because that is how you can hit the limits really, otherwise it's kinda hard to reach them, so in comparison it looks good. There are pros to using multiple modules (can load only one table when only one is needed, can have small files instead of few 10-20K LoC files), while cons are not manifested right now (as long as everything works and edit/purge doesn't feel slow, can say that there is no practical dimension for slowness).

        Loading editor
    • A FANDOM user
        Loading editor
Give Kudos to this message
You've given this message Kudos!
See who gave Kudos to this message