Merge branch 'develop' into fork_develop

Reviewed-on: https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/105
Co-authored-by: zaphkito <zaphkito@noreply.gitea.tendokyu.moe>
Co-committed-by: zaphkito <zaphkito@noreply.gitea.tendokyu.moe>

.gitattributes

@@ -0,0 +1,3 @@
+*.csv binary
+*.txt binary
+*.json binary

Dockerfile

@@ -1,6 +1,6 @@
 FROM python:3.9.15-slim-bullseye
 
-RUN apt update && apt install default-libmysqlclient-dev build-essential libtk nodejs npm -y
+RUN apt update && apt install default-libmysqlclient-dev build-essential libtk nodejs npm pkg-config -y
 
 WORKDIR /app
 COPY requirements.txt requirements.txt
@@ -12,10 +12,10 @@ RUN chmod +x entrypoint.sh
 
 COPY index.py index.py
 COPY dbutils.py dbutils.py
+COPY read.py read.py
 ADD core core
 ADD titles titles
-ADD config config
-ADD log log
+ADD logs logs
 ADD cert cert
 
 ENTRYPOINT [ "/app/entrypoint.sh" ]

LICENSE.txt

@@ -0,0 +1,13 @@
+           DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
+                   Version 2, December 2004
+ 
+Copyright (C) 2004 Sam Hocevar <sam@hocevar.net>
+
+Everyone is permitted to copy and distribute verbatim or modified
+copies of this license document, and changing it is allowed as long
+as the name is changed.
+ 
+           DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
+  TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+ 0. You just DO WHAT THE FUCK YOU WANT TO.

changelog.md

@@ -1,6 +1,120 @@
 # Changelog
 Documenting updates to ARTEMiS, to be updated every time the master branch is pushed to.
 
+## 20240302
+### SAO
++ Fixing new profile creation with right heroes and start VP
++ Fix to the Unanalyzed Log responses returning the wrong rewards
++ Documentation revised
+
+## 20240226
+### CXB
++ Fixing paths for rev.py
++ Changed encoding for handle_data_item_list_icon_request
+
+## 20240202
+### SAO
++ Added reader assets and edited the game specific documentation
+
+## 20240118
+### System
++ Added game version names to the readme
+
+## 20240109
+### System
++ Removed `ADD config config` from dockerfile [#83](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/83) (Thanks zaphkito!)
+
+### Aimedb
++ Fixed an error that resulted from trying to scan a banned or locked card
+
+## 20240108
+### System
++ Change how the underlying system handles URLs
+  + This can now allow for things like version-specific, or even keychip-specific URLs
+  + Specific changes to games are noted below
++ Fix docker files [#60](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/60) (Thanks Rylie!)
++ Fix support for python 3.8 - 3.10
+
+### Aimedb
++ Add support for SegaAuth key in games that support it (for now only Chunithm)
+  + This is a JWT that is sent to games, by Aimedb, that the games send to their game server, to verify that the access code the game is sending to the server was obtained via aimedb.
+  + Requires a base64-encoded secret to be set in the `core.yaml`
+
+### Chunithm
++ Fix Air support
++ Add saving for userRecentPlayerList
++ Add support for SegaAuthKey
++ Fix a bug arising if a user set their name to be 'true' or 'false'
++ Add support for Sun+ [#78](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/78) (Thanks EmmyHeart!)
++ Add `matching` section to `chuni.yaml`
++ ~~Change `udpHolePunchUri` and `reflectorUri` to be STUN and TURN servers~~ Reverted
++ Imrpove `GetGameSetting` request handling for different versions
++ Fix issue where songs would not always return all scores [#92](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/92) (Thanks Kumubou!)
+
+### maimai DX
++ Fix user charges failing to save
+
+### maimai
++ Made it functional
+
+### CXB
++ Improvements to request dispatching
++ Add support for non-omnimix music lists
+
+
+### IDZ
++ Fix news urls in accordance with the system change to URLs
+
+### Initial D THE ARCADE
++ Added support for Initial D THE ARCADE S2
+  + Story mode progress added
+  + Bunta Challenge/Touhou Project modes added
+  + Time Trials added
+  + Leaderboards added, but doesn't refresh sometimes
+  + Theory of Street mode added (with CPUs)
+  + Play Stamp/Timetrial events added
+  + Frontend to download profile added
+  + Importer to import profiles added
+
+### ONGEKI
++ Now supports HTTPS on a per-version basis
++ Merg PR [#61](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/61) (Thanks phantomlan!)
+  + Add Ranking Event Support
+  + Add reward list support
+  + Add version segregation to Event Ranking, Tech Challenge, and Music Ranking
+  + Now stores ClientTestmode and ClientSetting data
++ Fix mission points not adding correctly [#68](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/68) (Thanks phantomlan!)
++ Fix tech challenge [#70](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/70) (Thanks phantomlan!)
+
+### SAO
++ Change endpoint in accordance with the system change to URLs
++ Update request header class to be more accurate
++ Encrypted requests are now supported
++ Change to using handler classes instead of raw structs for simplicity
+
+### Wacca
++ Fix a server error causing a seperate error that casued issues
++ Add better error printing
++ Add better request validation
++ Fix HousingStartV2
++ Fix Lily's housing/get handler
+
+## 20231107
+### CXB
++  Hotfix `render_POST` sometimes failing to read the request body on large requests
+
+## 20231106
+### CXB
++ Hotfix `render_POST` function signature signature
++ Hotfix `handle_action_addenergy_request` hard failing if `get_energy` returns None
+
+## 20231015
+### maimai DX
++ Added support for FESTiVAL PLUS
+
+### Card Maker
++ Added support for maimai DX FESTiVAL PLUS
+
 ## 20230716
 ### General
 + Docker files added (#19)

core/__init__.py

@@ -1,6 +1,6 @@
 from core.config import CoreConfig
-from core.allnet import AllnetServlet
-from core.aimedb import AimedbFactory
+from core.allnet import AllnetServlet, BillingServlet
+from core.aimedb import AimedbServlette
 from core.title import TitleServlet
 from core.utils import Utils
 from core.mucha import MuchaServlet

core/adb_handlers/base.py

@@ -102,7 +102,7 @@ class ADBHeader:
         magic, protocol_ver, cmd, length, status, game_id, store_id, keychip_id = struct.unpack_from("<5H6sI12s", data)
         head = cls(magic, protocol_ver, cmd, length, status, game_id, store_id, keychip_id)
         
-        if head.length != len(data):
+        if head.length > len(data):
             raise ADBHeaderException(f"Length is incorrect! Expect {head.length}, got {len(data)}")
         
         return head

core/adb_handlers/felica.py

@@ -52,6 +52,7 @@ class ADBFelicaLookup2Response(ADBBaseResponse):
         self.access_code = access_code if access_code is not None else "00000000000000000000"
         self.company = CompanyCodes.SEGA
         self.portal_status = PortalRegStatus.NO_REG
+        self.auth_key = [0] * 256
 
     @classmethod
     def from_req(cls, req: ADBHeader, user_id: Union[int, None] = None, access_code: Union[str, None] = None) -> "ADBFelicaLookup2Response":
@@ -76,7 +77,7 @@ class ADBFelicaLookup2Response(ADBBaseResponse):
             access_code = bytes.fromhex(self.access_code),
             portal_status = self.portal_status.value,
             company_code = self.company.value,
-            auth_key = [0] * 256 # Unsupported
+            auth_key = self.auth_key
         ))
 
         self.head.length = HEADER_SIZE + len(resp_struct)

core/adb_handlers/lookup.py

@@ -52,6 +52,7 @@ class ADBLookupExResponse(ADBBaseResponse):
         super().__init__(code, length, status, game_id, store_id, keychip_id)
         self.user_id = user_id if user_id is not None else -1
         self.portal_reg = PortalRegStatus.NO_REG
+        self.auth_key = [0] * 256
 
     @classmethod
     def from_req(cls, req: ADBHeader, user_id: Union[int, None]) -> "ADBLookupExResponse":
@@ -72,7 +73,7 @@ class ADBLookupExResponse(ADBBaseResponse):
         body = resp_struct.build(dict(
             user_id = self.user_id,
             portal_reg = self.portal_reg.value,
-            auth_key = [0] * 256,
+            auth_key = self.auth_key,
             relation1 = -1,
             relation2 = -1
         ))

core/aimedb.py

@@ -1,24 +1,45 @@
-from twisted.internet.protocol import Factory, Protocol
 import logging, coloredlogs
 from Crypto.Cipher import AES
-import struct
-from typing import Dict, Tuple, Callable, Union
-from typing_extensions import Final
+from typing import Dict, Tuple, Callable, Union, Optional
+import asyncio
 from logging.handlers import TimedRotatingFileHandler
 
 from core.config import CoreConfig
+from core.utils import create_sega_auth_key
 from core.data import Data
 from .adb_handlers import *
 
-
-class AimedbProtocol(Protocol):
+class AimedbServlette():
     request_list: Dict[int, Tuple[Callable[[bytes, int], Union[ADBBaseResponse, bytes]], int, str]] = {}
-
     def __init__(self, core_cfg: CoreConfig) -> None:        
-        self.logger = logging.getLogger("aimedb")
         self.config = core_cfg        
         self.data = Data(core_cfg)
-        if core_cfg.aimedb.key == "":
+
+        self.logger = logging.getLogger("aimedb")
+        if not hasattr(self.logger, "initted"):
+            log_fmt_str = "[%(asctime)s] Aimedb | %(levelname)s | %(message)s"
+            log_fmt = logging.Formatter(log_fmt_str)
+
+            fileHandler = TimedRotatingFileHandler(
+                "{0}/{1}.log".format(self.config.server.log_dir, "aimedb"),
+                when="d",
+                backupCount=10,
+            )
+            fileHandler.setFormatter(log_fmt)
+
+            consoleHandler = logging.StreamHandler()
+            consoleHandler.setFormatter(log_fmt)
+
+            self.logger.addHandler(fileHandler)
+            self.logger.addHandler(consoleHandler)
+
+            self.logger.setLevel(self.config.aimedb.loglevel)
+            coloredlogs.install(
+                level=core_cfg.aimedb.loglevel, logger=self.logger, fmt=log_fmt_str
+            )
+            self.logger.initted = True
+
+        if not core_cfg.aimedb.key:
             self.logger.error("!!!KEY NOT SET!!!")
             exit(1)
 
@@ -39,27 +60,31 @@ class AimedbProtocol(Protocol):
 
         self.register_handler(0x13, 0x14, self.handle_log_ex, 'aime_log_ex')
         self.register_handler(0x64, 0x65, self.handle_hello, 'hello')
-        self.register_handler(0x66, 0, self.handle_goodbye, 'goodbye')
 
     def register_handler(self, cmd: int, resp:int, handler: Callable[[bytes, int], Union[ADBBaseResponse, bytes]], name: str) -> None:
         self.request_list[cmd] = (handler, resp, name)
     
-    def append_padding(self, data: bytes):
-        """Appends 0s to the end of the data until it's at the correct size"""
-        length = struct.unpack_from("<H", data, 6)
-        padding_size = length[0] - len(data)
-        data += bytes(padding_size)
-        return data
+    def start(self) -> None:
+        self.logger.info(f"Start on port {self.config.aimedb.port}")
+        addr = self.config.aimedb.listen_address if self.config.aimedb.listen_address else self.config.server.listen_address
+        asyncio.create_task(asyncio.start_server(self.dataReceived, addr, self.config.aimedb.port))
     
-    def connectionMade(self) -> None:
-        self.logger.debug(f"{self.transport.getPeer().host} Connected")
+    async def dataReceived(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
+        self.logger.debug(f"Connection made from {writer.get_extra_info('peername')[0]}")
+        while True:
+            try:
+                data: bytes = await reader.read(4096)
+                if len(data) == 0:
+                    self.logger.debug("Connection closed")
+                    return
+                await self.process_data(data, reader, writer)
+                await writer.drain()
+            except ConnectionResetError as e:
+                self.logger.debug("Connection reset, disconnecting")
+                return
 
-    def connectionLost(self, reason) -> None:
-        self.logger.debug(
-            f"{self.transport.getPeer().host} Disconnected - {reason.value}"
-        )
-
-    def dataReceived(self, data: bytes) -> None:
+    async def process_data(self, data: bytes, reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> Optional[bytes]:
+        addr = writer.get_extra_info('peername')[0]
         cipher = AES.new(self.config.aimedb.key.encode(), AES.MODE_ECB)
 
         try:
@@ -67,9 +92,9 @@ class AimedbProtocol(Protocol):
         
         except Exception as e:
             self.logger.error(f"Failed to decrypt {data.hex()} because {e}")
-            return None
+            return
 
-        self.logger.debug(f"{self.transport.getPeer().host} wrote {decrypted.hex()}")
+        self.logger.debug(f"{addr} wrote {decrypted.hex()}")
 
         try:
             head = ADBHeader.from_data(decrypted)
@@ -78,7 +103,9 @@ class AimedbProtocol(Protocol):
             self.logger.error(f"Error parsing ADB header: {e}")
             try:    
                 encrypted = cipher.encrypt(ADBBaseResponse().make())
-                self.transport.write(encrypted)
+                writer.write(encrypted)
+                await writer.drain()
+                return
 
             except Exception as e:
                 self.logger.error(f"Failed to encrypt default response because {e}")
@@ -88,46 +115,51 @@ class AimedbProtocol(Protocol):
         if head.keychip_id == "ABCD1234567" or head.store_id == 0xfff0:
             self.logger.warning(f"Request from uninitialized AMLib: {vars(head)}")
 
+        if head.cmd == 0x66:
+            self.logger.info("Goodbye")
+            writer.close()
+            return
+
         handler, resp_code, name = self.request_list.get(head.cmd, (self.handle_default, None, 'default'))
 
         if resp_code is None:
             self.logger.warning(f"No handler for cmd {hex(head.cmd)}")
         
         elif resp_code > 0:
-            self.logger.info(f"{name} from {head.keychip_id} ({head.game_id}) @ {self.transport.getPeer().host}")
+            self.logger.info(f"{name} from {head.keychip_id} ({head.game_id}) @ {addr}")
         
-        resp = handler(decrypted, resp_code)
+        resp = await handler(decrypted, resp_code)
 
         if type(resp) == ADBBaseResponse or issubclass(type(resp), ADBBaseResponse):
             resp_bytes = resp.make()
-            if len(resp_bytes) != resp.head.length:
-                resp_bytes = self.append_padding(resp_bytes)
 
         elif type(resp) == bytes:
             resp_bytes = resp
         
         elif resp is None: # Nothing to send, probably a goodbye
+            self.logger.warn(f"None return by handler for {name}")
             return
         
         else:
+            self.logger.error(f"Unsupported type returned by ADB handler for {name}: {type(resp)}")
             raise TypeError(f"Unsupported type returned by ADB handler for {name}: {type(resp)}")
 
         try:
             encrypted = cipher.encrypt(resp_bytes)
             self.logger.debug(f"Response {resp_bytes.hex()}")
-            self.transport.write(encrypted)
+            writer.write(encrypted)
 
         except Exception as e:
             self.logger.error(f"Failed to encrypt {resp_bytes.hex()} because {e}")
     
-    def handle_default(self, data: bytes, resp_code: int, length: int = 0x20) -> ADBBaseResponse:
+    async def handle_default(self, data: bytes, resp_code: int, length: int = 0x20) -> ADBBaseResponse:
         req = ADBHeader.from_data(data)
         return ADBBaseResponse(resp_code, length, 1, req.game_id, req.store_id, req.keychip_id, req.protocol_ver)
 
-    def handle_hello(self, data: bytes, resp_code: int) -> ADBBaseResponse:
-        return self.handle_default(data, resp_code)
+    async def handle_hello(self, data: bytes, resp_code: int) -> ADBBaseResponse:
+        return await self.handle_default(data, resp_code)
 
-    def handle_campaign(self, data: bytes, resp_code: int) -> ADBBaseResponse:
+    async def handle_campaign(self, data: bytes, resp_code: int) -> ADBBaseResponse:
         h = ADBHeader.from_data(data)
         if h.protocol_ver >= 0x3030:
             req = h
@@ -142,29 +174,60 @@ class AimedbProtocol(Protocol):
         # We don't currently support campaigns
         return resp
 
-    def handle_lookup(self, data: bytes, resp_code: int) -> ADBBaseResponse:
+    async def handle_lookup(self, data: bytes, resp_code: int) -> ADBBaseResponse:
         req = ADBLookupRequest(data)
-        user_id = self.data.card.get_user_id_from_card(req.access_code)
+        user_id = await self.data.card.get_user_id_from_card(req.access_code)
+        is_banned = await self.data.card.get_card_banned(req.access_code)
+        is_locked = await self.data.card.get_card_locked(req.access_code)
         
         ret = ADBLookupResponse.from_req(req.head, user_id)
+        if is_banned and is_locked:
+            ret.head.status = ADBStatus.BAN_SYS_USER
+        elif is_banned:
+            ret.head.status = ADBStatus.BAN_SYS
+        elif is_locked:
+            ret.head.status = ADBStatus.LOCK_USER
         
         self.logger.info(
             f"access_code {req.access_code} -> user_id {ret.user_id}"
         )
+        
+        if user_id and user_id > 0:
+            await self.data.card.update_card_last_login(req.access_code)
         return ret
 
-    def handle_lookup_ex(self, data: bytes, resp_code: int) -> ADBBaseResponse:
+    async def handle_lookup_ex(self, data: bytes, resp_code: int) -> ADBBaseResponse:
         req = ADBLookupRequest(data)
-        user_id = self.data.card.get_user_id_from_card(req.access_code)
+        user_id = await self.data.card.get_user_id_from_card(req.access_code)
+
+        is_banned = await self.data.card.get_card_banned(req.access_code)
+        is_locked = await self.data.card.get_card_locked(req.access_code)
 
         ret = ADBLookupExResponse.from_req(req.head, user_id)
+        if is_banned and is_locked:
+            ret.head.status = ADBStatus.BAN_SYS_USER
+        elif is_banned:
+            ret.head.status = ADBStatus.BAN_SYS
+        elif is_locked:
+            ret.head.status = ADBStatus.LOCK_USER
 
         self.logger.info(
             f"access_code {req.access_code} -> user_id {ret.user_id}"
         )
+
+        if user_id and user_id > 0 and self.config.aimedb.id_secret:
+            auth_key = create_sega_auth_key(user_id, req.head.game_id, req.head.store_id, req.head.keychip_id, self.config.aimedb.id_secret, self.config.aimedb.id_lifetime_seconds)
+            if auth_key is not None:
+                auth_key_extra_len = 256 - len(auth_key)
+                auth_key_full = auth_key.encode() + (b"\0" * auth_key_extra_len)
+                self.logger.debug(f"Generated auth token {auth_key}")
+                ret.auth_key = auth_key_full
+
+        if user_id and user_id > 0:
+            await self.data.card.update_card_last_login(req.access_code)
         return ret
 
-    def handle_felica_lookup(self, data: bytes, resp_code: int) -> bytes:
+    async def handle_felica_lookup(self, data: bytes, resp_code: int) -> bytes:
         """
         On official, I think a card has to be registered for this to actually work, but 
         I'm making the executive decision to not implement that and just kick back our
@@ -180,7 +243,7 @@ class AimedbProtocol(Protocol):
         )
         return ADBFelicaLookupResponse.from_req(req.head, ac)
 
-    def handle_felica_register(self, data: bytes, resp_code: int) -> bytes:
+    async def handle_felica_register(self, data: bytes, resp_code: int) -> bytes:
         """
         I've never seen this used.
         """
@@ -188,14 +251,14 @@ class AimedbProtocol(Protocol):
         ac = self.data.card.to_access_code(req.idm)
         
         if self.config.server.allow_user_registration:
-            user_id = self.data.user.create_user()
+            user_id = await self.data.user.create_user()
 
             if user_id is None:
                 self.logger.error("Failed to register user!")
                 user_id = -1
 
             else:
-                card_id = self.data.card.create_card(user_id, ac)
+                card_id = await self.data.card.create_card(user_id, ac)
 
                 if card_id is None:
                     self.logger.error("Failed to register card!")
@@ -210,12 +273,14 @@ class AimedbProtocol(Protocol):
                 f"Registration blocked!: access code {ac} (IDm: {req.idm} PMm: {req.pmm})"
             )
 
+        if user_id > 0:
+            await self.data.card.update_card_last_login(ac)
         return ADBFelicaLookupResponse.from_req(req.head, ac)
 
-    def handle_felica_lookup_ex(self, data: bytes, resp_code: int) -> bytes:
+    async def handle_felica_lookup_ex(self, data: bytes, resp_code: int) -> bytes:
         req = ADBFelicaLookup2Request(data)
         access_code = self.data.card.to_access_code(req.idm)
-        user_id = self.data.card.get_user_id_from_card(access_code=access_code)
+        user_id = await self.data.card.get_user_id_from_card(access_code=access_code)
 
         if user_id is None:
             user_id = -1
@@ -224,9 +289,21 @@ class AimedbProtocol(Protocol):
             f"idm {req.idm} ipm {req.pmm} -> access_code {access_code} user_id {user_id}"
         )
 
-        return ADBFelicaLookup2Response.from_req(req.head, user_id, access_code)
+        resp = ADBFelicaLookup2Response.from_req(req.head, user_id, access_code)
 
-    def handle_campaign_clear(self, data: bytes, resp_code: int) -> ADBBaseResponse:
+        if user_id and user_id > 0 and self.config.aimedb.id_secret:
+            auth_key = create_sega_auth_key(user_id, req.head.game_id, req.head.store_id, req.head.keychip_id, self.config.aimedb.id_secret, self.config.aimedb.id_lifetime_seconds)
+            if auth_key is not None:
+                auth_key_extra_len = 256 - len(auth_key)
+                auth_key_full = auth_key.encode() + (b"\0" * auth_key_extra_len)
+                self.logger.debug(f"Generated auth token {auth_key}")
+                resp.auth_key = auth_key_full
+        
+        if user_id and user_id > 0:
+            await self.data.card.update_card_last_login(access_code)
+        return resp
+
+    async def handle_campaign_clear(self, data: bytes, resp_code: int) -> ADBBaseResponse:
         req = ADBCampaignClearRequest(data)
 
         resp = ADBCampaignClearResponse.from_req(req.head)
@@ -234,19 +311,19 @@ class AimedbProtocol(Protocol):
         # We don't support campaign stuff
         return resp
 
-    def handle_register(self, data: bytes, resp_code: int) -> bytes:
+    async def handle_register(self, data: bytes, resp_code: int) -> bytes:
         req = ADBLookupRequest(data)
         user_id = -1
 
         if self.config.server.allow_user_registration:
-            user_id = self.data.user.create_user()
+            user_id = await self.data.user.create_user()
 
             if user_id is None:
                 self.logger.error("Failed to register user!")
                 user_id = -1
 
             else:
-                card_id = self.data.card.create_card(user_id, req.access_code)
+                card_id = await self.data.card.create_card(user_id, req.access_code)
 
                 if card_id is None:
                     self.logger.error("Failed to register card!")
@@ -265,20 +342,23 @@ class AimedbProtocol(Protocol):
         if resp.user_id <= 0:
             resp.head.status = ADBStatus.BAN_SYS # Closest we can get to a "You cannot register"
 
+        else:
+            await self.data.card.update_card_last_login(req.access_code)
+
         return resp
 
     # TODO: Save these in some capacity, as deemed relevant
-    def handle_status_log(self, data: bytes, resp_code: int) -> bytes:
+    async def handle_status_log(self, data: bytes, resp_code: int) -> bytes:
         req = ADBStatusLogRequest(data)
         self.logger.info(f"User {req.aime_id} logged {req.status.name} event")
         return ADBBaseResponse(resp_code, 0x20, 1, req.head.game_id, req.head.store_id, req.head.keychip_id, req.head.protocol_ver)
 
-    def handle_log(self, data: bytes, resp_code: int) -> bytes:
+    async def handle_log(self, data: bytes, resp_code: int) -> bytes:
         req = ADBLogRequest(data)
         self.logger.info(f"User {req.aime_id} logged {req.status.name} event, credit_ct: {req.credit_ct} bet_ct: {req.bet_ct} won_ct: {req.won_ct}")
         return ADBBaseResponse(resp_code, 0x20, 1, req.head.game_id, req.head.store_id, req.head.keychip_id, req.head.protocol_ver)
 
-    def handle_log_ex(self, data: bytes, resp_code: int) -> bytes:
+    async def handle_log_ex(self, data: bytes, resp_code: int) -> bytes:
         req = ADBLogExRequest(data)
         strs = []
         self.logger.info(f"Recieved {req.num_logs} or {len(req.logs)} logs")
@@ -287,43 +367,3 @@ class AimedbProtocol(Protocol):
             self.logger.debug(f"User {req.logs[x].aime_id} logged {req.logs[x].status.name} event, credit_ct: {req.logs[x].credit_ct} bet_ct: {req.logs[x].bet_ct} won_ct: {req.logs[x].won_ct}")
         return ADBLogExResponse.from_req(req.head)
 
-    def handle_goodbye(self, data: bytes, resp_code: int) -> None:
-        self.logger.info(f"goodbye from {self.transport.getPeer().host}")
-        self.transport.loseConnection()
-        return
-
-class AimedbFactory(Factory):
-    protocol = AimedbProtocol
-
-    def __init__(self, cfg: CoreConfig) -> None:
-        self.config = cfg
-        log_fmt_str = "[%(asctime)s] Aimedb | %(levelname)s | %(message)s"
-        log_fmt = logging.Formatter(log_fmt_str)
-        self.logger = logging.getLogger("aimedb")
-
-        fileHandler = TimedRotatingFileHandler(
-            "{0}/{1}.log".format(self.config.server.log_dir, "aimedb"),
-            when="d",
-            backupCount=10,
-        )
-        fileHandler.setFormatter(log_fmt)
-
-        consoleHandler = logging.StreamHandler()
-        consoleHandler.setFormatter(log_fmt)
-
-        self.logger.addHandler(fileHandler)
-        self.logger.addHandler(consoleHandler)
-
-        self.logger.setLevel(self.config.aimedb.loglevel)
-        coloredlogs.install(
-            level=cfg.aimedb.loglevel, logger=self.logger, fmt=log_fmt_str
-        )
-
-        if self.config.aimedb.key == "":
-            self.logger.error("Please set 'key' field in your config file.")
-            exit(1)
-
-        self.logger.info(f"Ready on port {self.config.aimedb.port}")
-
-    def buildProtocol(self, addr):
-        return AimedbProtocol(self.config)

core/allnet.py

@@ -1,25 +1,30 @@
-from typing import Dict, List, Any, Optional, Tuple, Union, Final
-import logging, coloredlogs
-from logging.handlers import TimedRotatingFileHandler
-from twisted.web.http import Request
-from datetime import datetime
 import pytz
 import base64
 import zlib
 import json
+import yaml
+import logging
+import coloredlogs
+import urllib.parse
+import math
+from typing import Dict, List, Any, Optional, Union, Final
+from logging.handlers import TimedRotatingFileHandler
+from starlette.requests import Request
+from starlette.responses import PlainTextResponse
+from starlette.applications import Starlette
+from starlette.routing import Route
+from datetime import datetime
 from enum import Enum
 from Crypto.PublicKey import RSA
 from Crypto.Hash import SHA
 from Crypto.Signature import PKCS1_v1_5
-from time import strptime
-from os import path
-import urllib.parse
-import math
+from os import path, environ, mkdir, access, W_OK
 
-from core.config import CoreConfig
-from core.utils import Utils
-from core.data import Data
-from core.const import *
+from .config import CoreConfig
+from .utils import Utils
+from .data import Data
+from .const import *
+from .title import TitleServlet
 
 BILLING_DT_FORMAT: Final[str] = "%Y%m%d%H%M%S"
 
@@ -33,13 +38,67 @@ class ALLNET_STAT(Enum):
     bad_machine = -2
     bad_shop = -3
 
+class DLI_STATUS(Enum):
+    START = 0
+    GET_DOWNLOAD_CONFIGURATION = 1
+    WAIT_DOWNLOAD = 2
+    DOWNLOADING = 3
+
+    NOT_SPECIFY_DLI = 100
+    ONLY_POST_REPORT = 101
+    STOPPED_BY_APP_RELEASE = 102
+    STOPPED_BY_OPT_RELEASE = 103
+
+    DOWNLOAD_COMPLETE_RECENTLY = 110
+
+    DOWNLOAD_COMPLETE_WAIT_RELEASE_TIME = 120
+    DOWNLOAD_COMPLETE_BUT_NOT_SYNC_SERVER = 121
+    DOWNLOAD_COMPLETE_BUT_NOT_FIRST_RESUME = 122
+    DOWNLOAD_COMPLETE_BUT_NOT_FIRST_LAUNCH = 123
+    DOWNLOAD_COMPLETE_WAIT_UPDATE = 124
+
+    DOWNLOAD_COMPLETE_AND_ALREADY_UPDATE = 130
+
+    ERROR_AUTH_FAILURE = 200
+
+    ERROR_GET_DLI_HTTP = 300
+    ERROR_GET_DLI = 301
+    ERROR_PARSE_DLI = 302
+    ERROR_INVALID_GAME_ID = 303
+    ERROR_INVALID_IMAGE_LIST = 304
+    ERROR_GET_DLI_APP = 305
+
+    ERROR_GET_BOOT_ID = 400
+    ERROR_ACCESS_SERVER = 401
+    ERROR_NO_IMAGE = 402
+    ERROR_ACCESS_IMAGE = 403
+
+    ERROR_DOWNLOAD_APP = 500
+    ERROR_DOWNLOAD_OPT = 501
+
+    ERROR_DISK_FULL = 600
+    ERROR_UNINSTALL = 601
+    ERROR_INSTALL_APP = 602
+    ERROR_INSTALL_OPT = 603
+
+    ERROR_GET_DLI_INTERNAL = 900
+    ERROR_ICF = 901
+    ERROR_CHECK_RELEASE_INTERNAL = 902
+    UNKNOWN = 999 # Not the actual enum val but it needs to be here as a catch-all
+
+    @classmethod
+    def from_int(cls, num: int) -> "DLI_STATUS":
+        try:
+            return cls(num)
+        except ValueError:
+            return cls.UNKNOWN
+
 class AllnetServlet:
+    allnet_registry: Dict[str, Any] = {}
     def __init__(self, core_cfg: CoreConfig, cfg_folder: str):
-        super().__init__()
         self.config = core_cfg
         self.config_folder = cfg_folder
         self.data = Data(core_cfg)
-        self.uri_registry: Dict[str, Tuple[str, str]] = {}
 
         self.logger = logging.getLogger("allnet")
         if not hasattr(self.logger, "initialized"):
@@ -65,35 +124,22 @@ class AllnetServlet:
             )
             self.logger.initialized = True
 
-        plugins = Utils.get_all_titles()
+    def startup(self) -> None:
+        self.logger.info(f"Ready on port {self.config.allnet.port if self.config.allnet.standalone else self.config.server.port}")
+        if not TitleServlet.title_registry:
+            TitleServlet(self.config, self.config_folder)
 
-        if len(plugins) == 0:
-            self.logger.error("No games detected!")
-
-        for _, mod in plugins.items():
-            if hasattr(mod, "index") and hasattr(mod.index, "get_allnet_info"):
-                for code in mod.game_codes:
-                    enabled, uri, host = mod.index.get_allnet_info(
-                        code, self.config, self.config_folder
-                    )
-
-                    if enabled:
-                        self.uri_registry[code] = (uri, host)
-
-        self.logger.info(
-            f"Serving {len(self.uri_registry)} game codes port {core_cfg.allnet.port}"
-        )
-
-    def handle_poweron(self, request: Request, _: Dict):
+    async def handle_poweron(self, request: Request):
         request_ip = Utils.get_ip_addr(request)
-        pragma_header = request.getHeader('Pragma')
+        pragma_header = request.headers.get('Pragma', "")
         is_dfi = pragma_header is not None and pragma_header == "DFI"
+        data = await request.body()
         
         try:
             if is_dfi:
-                req_urlencode = self.from_dfi(request.content.getvalue())
+                req_urlencode = self.from_dfi(data)
             else:
-                req_urlencode = request.content.getvalue().decode()
+                req_urlencode = data
 
             req_dict = self.allnet_req_to_dict(req_urlencode)
             if req_dict is None:
@@ -110,7 +156,7 @@ class AllnetServlet:
         except AllnetRequestException as e:
             if e.message != "":
                 self.logger.error(e)
-            return b""
+            return PlainTextResponse()
 
         if req.format_ver == 3:
             resp = AllnetPowerOnResponse3(req.token)
@@ -121,42 +167,42 @@ class AllnetServlet:
 
         self.logger.debug(f"Allnet request: {vars(req)}")
 
-        machine = self.data.arcade.get_machine(req.serial)        
+        machine = await self.data.arcade.get_machine(req.serial)        
         if machine is None and not self.config.server.allow_unregistered_serials:
             msg = f"Unrecognised serial {req.serial} attempted allnet auth from {request_ip}."
-            self.data.base.log_event(
+            await self.data.base.log_event(
                 "allnet", "ALLNET_AUTH_UNKNOWN_SERIAL", logging.WARN, msg
             )
             self.logger.warning(msg)
 
             resp.stat = ALLNET_STAT.bad_machine.value
             resp_dict = {k: v for k, v in vars(resp).items() if v is not None}
-            return (urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n").encode("utf-8")
+            return PlainTextResponse(urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n")
 
         if machine is not None:
-            arcade = self.data.arcade.get_arcade(machine["arcade"])
+            arcade = await self.data.arcade.get_arcade(machine["arcade"])
             if self.config.server.check_arcade_ip:
                 if arcade["ip"] and arcade["ip"] is not None and arcade["ip"] != req.ip:
                     msg = f"Serial {req.serial} attempted allnet auth from bad IP {req.ip} (expected {arcade['ip']})."
-                    self.data.base.log_event(
+                    await self.data.base.log_event(
                         "allnet", "ALLNET_AUTH_BAD_IP", logging.ERROR, msg
                     )
                     self.logger.warning(msg)
 
                     resp.stat = ALLNET_STAT.bad_shop.value
                     resp_dict = {k: v for k, v in vars(resp).items() if v is not None}
-                    return (urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n").encode("utf-8")
+                    return PlainTextResponse(urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n")
                 
-                elif not arcade["ip"] or arcade["ip"] is None and self.config.server.strict_ip_checking:
+                elif (not arcade["ip"] or arcade["ip"] is None) and self.config.server.strict_ip_checking:
                     msg = f"Serial {req.serial} attempted allnet auth from bad IP {req.ip}, but arcade {arcade['id']} has no IP set! (strict checking enabled)."
-                    self.data.base.log_event(
+                    await self.data.base.log_event(
                         "allnet", "ALLNET_AUTH_NO_SHOP_IP", logging.ERROR, msg
                     )
                     self.logger.warning(msg)
 
                     resp.stat = ALLNET_STAT.bad_shop.value
                     resp_dict = {k: v for k, v in vars(resp).items() if v is not None}
-                    return (urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n").encode("utf-8")
+                    return PlainTextResponse(urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n")
 
 
             country = (
@@ -166,7 +212,7 @@ class AllnetServlet:
                 country = AllnetCountryCode.JAPAN.value
 
             resp.country = country
-            resp.place_id = arcade["id"]
+            resp.place_id = f"{arcade['id']:04X}"
             resp.allnet_id = machine["id"]
             resp.name = arcade["name"] if arcade["name"] is not None else ""
             resp.nickname = arcade["nickname"] if arcade["nickname"] is not None else ""
@@ -176,53 +222,57 @@ class AllnetServlet:
                 else AllnetJapanRegionId.AICHI.value
             )
             resp.region_name0 = (
-                arcade["country"]
-                if arcade["country"] is not None
-                else AllnetCountryCode.JAPAN.value
-            )
-            resp.region_name1 = (
                 arcade["state"]
                 if arcade["state"] is not None
                 else AllnetJapanRegionId.AICHI.name
             )
+            resp.region_name1 = (
+                arcade["country"]
+                if arcade["country"] is not None
+                else AllnetCountryCode.JAPAN.value
+            )
             resp.region_name2 = arcade["city"] if arcade["city"] is not None else ""
-            resp.client_timezone = (
-                arcade["timezone"] if arcade["timezone"] is not None else "+0900"
+            resp.client_timezone = ( # lmao
+                arcade["timezone"] if arcade["timezone"] is not None else "+0900" if req.format_ver == 3 else "+09:00"
             )
         
-        if req.game_id not in self.uri_registry:
+        if req.game_id not in TitleServlet.title_registry:
             if not self.config.server.is_develop:
                 msg = f"Unrecognised game {req.game_id} attempted allnet auth from {request_ip}."
-                self.data.base.log_event(
+                await self.data.base.log_event(
                     "allnet", "ALLNET_AUTH_UNKNOWN_GAME", logging.WARN, msg
                 )
                 self.logger.warning(msg)
 
                 resp.stat = ALLNET_STAT.bad_game.value
                 resp_dict = {k: v for k, v in vars(resp).items() if v is not None}
-                return (urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n").encode("utf-8")
+                return PlainTextResponse(urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n")
 
             else:
                 self.logger.info(
                     f"Allowed unknown game {req.game_id} v{req.ver} to authenticate from {request_ip} due to 'is_develop' being enabled. S/N: {req.serial}"
                 )
-                resp.uri = f"http://{self.config.title.hostname}:{self.config.title.port}/{req.game_id}/{req.ver.replace('.', '')}/"
-                resp.host = f"{self.config.title.hostname}:{self.config.title.port}"
+                resp.uri = f"http://{self.config.server.hostname}:{self.config.server.port}/{req.game_id}/{req.ver.replace('.', '')}/"
+                resp.host = f"{self.config.server.hostname}:{self.config.server.port}"
                 
                 resp_dict = {k: v for k, v in vars(resp).items() if v is not None}
                 resp_str = urllib.parse.unquote(urllib.parse.urlencode(resp_dict))
                 
                 self.logger.debug(f"Allnet response: {resp_str}")
-                return (resp_str + "\n").encode("utf-8")
+                return PlainTextResponse(resp_str + "\n")
 
-        resp.uri, resp.host = self.uri_registry[req.game_id]
         
         int_ver = req.ver.replace(".", "")
-        resp.uri = resp.uri.replace("$v", int_ver)
-        resp.host = resp.host.replace("$v", int_ver)
+        try:
+            resp.uri, resp.host = TitleServlet.title_registry[req.game_id].get_allnet_info(req.game_id, int(int_ver), req.serial)
+        except Exception as e:
+            self.logger.error(f"Error running get_allnet_info for {req.game_id} - {e}")
+            resp.stat = ALLNET_STAT.bad_game.value
+            resp_dict = {k: v for k, v in vars(resp).items() if v is not None}
+            return PlainTextResponse(urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n")
 
         msg = f"{req.serial} authenticated from {request_ip}: {req.game_id} v{req.ver}"
-        self.data.base.log_event("allnet", "ALLNET_AUTH_SUCCESS", logging.INFO, msg)
+        await self.data.base.log_event("allnet", "ALLNET_AUTH_SUCCESS", logging.INFO, msg)
         self.logger.info(msg)
 
         resp_dict = {k: v for k, v in vars(resp).items() if v is not None}
@@ -234,18 +284,19 @@ class AllnetServlet:
             request.responseHeaders.addRawHeader('Pragma', 'DFI')
             return self.to_dfi(resp_str)"""
 
-        return resp_str.encode("utf-8")
+        return PlainTextResponse(resp_str)
 
-    def handle_dlorder(self, request: Request, _: Dict):
+    async def handle_dlorder(self, request: Request):
         request_ip = Utils.get_ip_addr(request)
-        pragma_header = request.getHeader('Pragma')
+        pragma_header = request.headers.get('Pragma', "")
         is_dfi = pragma_header is not None and pragma_header == "DFI"
+        data = await request.body()
         
         try:
             if is_dfi:
-                req_urlencode = self.from_dfi(request.content.getvalue())
+                req_urlencode = self.from_dfi(data)
             else:
-                req_urlencode = request.content.getvalue().decode()
+                req_urlencode = data.decode()
 
             req_dict = self.allnet_req_to_dict(req_urlencode)
             if req_dict is None:
@@ -262,7 +313,7 @@ class AllnetServlet:
         except AllnetRequestException as e:
             if e.message != "":
                 self.logger.error(e)
-            return b""
+            return PlainTextResponse()
 
         self.logger.info(
             f"DownloadOrder from {request_ip} -> {req.game_id} v{req.ver} serial {req.serial}"
@@ -273,54 +324,54 @@ class AllnetServlet:
             not self.config.allnet.allow_online_updates
             or not self.config.allnet.update_cfg_folder
         ):
-            return urllib.parse.unquote(urllib.parse.urlencode(vars(resp))) + "\n"
+            return PlainTextResponse(urllib.parse.unquote(urllib.parse.urlencode(vars(resp))) + "\n")
 
         else:  # TODO: Keychip check
             if path.exists(
                 f"{self.config.allnet.update_cfg_folder}/{req.game_id}-{req.ver.replace('.', '')}-app.ini"
             ):
-                resp.uri = f"http://{self.config.title.hostname}:{self.config.title.port}/dl/ini/{req.game_id}-{req.ver.replace('.', '')}-app.ini"
+                resp.uri = f"http://{self.config.server.hostname}:{self.config.server.port}/dl/ini/{req.game_id}-{req.ver.replace('.', '')}-app.ini"
 
             if path.exists(
                 f"{self.config.allnet.update_cfg_folder}/{req.game_id}-{req.ver.replace('.', '')}-opt.ini"
             ):
-                resp.uri += f"|http://{self.config.title.hostname}:{self.config.title.port}/dl/ini/{req.game_id}-{req.ver.replace('.', '')}-opt.ini"
+                resp.uri += f"|http://{self.config.server.hostname}:{self.config.server.port}/dl/ini/{req.game_id}-{req.ver.replace('.', '')}-opt.ini"
 
             self.logger.debug(f"Sending download uri {resp.uri}")
-            self.data.base.log_event("allnet", "DLORDER_REQ_SUCCESS", logging.INFO, f"{Utils.get_ip_addr(request)} requested DL Order for {req.serial} {req.game_id} v{req.ver}")
+            await self.data.base.log_event("allnet", "DLORDER_REQ_SUCCESS", logging.INFO, f"{Utils.get_ip_addr(request)} requested DL Order for {req.serial} {req.game_id} v{req.ver}")
 
             res_str = urllib.parse.unquote(urllib.parse.urlencode(vars(resp))) + "\n"
             """if is_dfi:
                 request.responseHeaders.addRawHeader('Pragma', 'DFI')
                 return self.to_dfi(res_str)"""
 
-            return res_str
+            return PlainTextResponse(res_str)
 
+    async def handle_dlorder_ini(self, request: Request) -> bytes:
+        req_file = request.path_params.get("file", "").replace("%0A", "").replace("\n", "")
 
-    def handle_dlorder_ini(self, request: Request, match: Dict) -> bytes:
-        if "file" not in match:
-            return b""
-
-        req_file = match["file"].replace("%0A", "")
+        if not req_file:
+            return PlainTextResponse(status_code=404)
 
         if path.exists(f"{self.config.allnet.update_cfg_folder}/{req_file}"):
             self.logger.info(f"Request for DL INI file {req_file} from {Utils.get_ip_addr(request)} successful")
-            self.data.base.log_event("allnet", "DLORDER_INI_SENT", logging.INFO, f"{Utils.get_ip_addr(request)} successfully recieved {req_file}")
+            await self.data.base.log_event("allnet", "DLORDER_INI_SENT", logging.INFO, f"{Utils.get_ip_addr(request)} successfully recieved {req_file}")
             
-            return open(
-                f"{self.config.allnet.update_cfg_folder}/{req_file}", "rb"
-            ).read()
+            return PlainTextResponse(open(
+                f"{self.config.allnet.update_cfg_folder}/{req_file}", "r", encoding="utf-8"
+            ).read())
 
         self.logger.info(f"DL INI File {req_file} not found")
-        return b""
+        return PlainTextResponse()
 
-    def handle_dlorder_report(self, request: Request, match: Dict) -> bytes:
-        req_raw = request.content.getvalue()
+    async def handle_dlorder_report(self, request: Request) -> bytes:
+        req_raw = await request.body()
+        client_ip = Utils.get_ip_addr(request)
         try:
             req_dict: Dict = json.loads(req_raw)
         except Exception as e:
             self.logger.warning(f"Failed to parse DL Report: {e}")
-            return "NG"
+            return PlainTextResponse("NG")
         
         dl_data_type = DLIMG_TYPE.app
         dl_data = req_dict.get("appimage", {})
@@ -331,18 +382,24 @@ class AllnetServlet:
         
         if dl_data is None or not dl_data:
             self.logger.warning(f"Failed to parse DL Report: Invalid format - contains neither appimage nor optimage")
-            return "NG"
+            return PlainTextResponse("NG")
 
-        dl_report_data = DLReport(dl_data, dl_data_type)
+        rep = DLReport(dl_data, dl_data_type)
 
-        if not dl_report_data.validate():
-            self.logger.warning(f"Failed to parse DL Report: Invalid format - {dl_report_data.err}")
-            return "NG"
+        if not rep.validate():
+            self.logger.warning(f"Failed to parse DL Report: Invalid format - {rep.err}")
+            return PlainTextResponse("NG")
         
-        return "OK"
+        msg = f"{rep.serial} @ {client_ip} reported {rep.rep_type.name} download state {rep.rf_state.name} for {rep.gd} v{rep.dav}:"\
+              f" {rep.tdsc}/{rep.tsc} segments downloaded for working files {rep.wfl} with {rep.dfl if rep.dfl else 'none'} complete."
         
-    def handle_loaderstaterecorder(self, request: Request, match: Dict) -> bytes:
-        req_data = request.content.getvalue()
+        await self.data.base.log_event("allnet", "DL_REPORT", logging.INFO, msg, dl_data)
+        self.logger.info(msg)
+
+        return PlainTextResponse("OK")
+
+    async def handle_loaderstaterecorder(self, request: Request) -> bytes:
+        req_data = await request.body()
         sections = req_data.decode("utf-8").split("\r\n")
         
         req_dict = dict(urllib.parse.parse_qsl(sections[0]))
@@ -354,129 +411,17 @@ class AllnetServlet:
         ip = Utils.get_ip_addr(request)
 
         if serial is None or num_files_dld is None or num_files_to_dl is None or dl_state is None:
-            return "NG".encode()
+            return PlainTextResponse("NG")
 
         self.logger.info(f"LoaderStateRecorder Request from {ip} {serial}: {num_files_dld}/{num_files_to_dl} Files download (State: {dl_state})")
-        return "OK".encode()
+        return PlainTextResponse("OK")
     
-    def handle_alive(self, request: Request, match: Dict) -> bytes:
-        return "OK".encode()
+    async def handle_alive(self, request: Request) -> bytes:
+        return PlainTextResponse("OK")
 
-    def handle_billing_request(self, request: Request, _: Dict):
-        req_raw = request.content.getvalue()
-        
-        if request.getHeader('Content-Type') == "application/octet-stream":
-            req_unzip = zlib.decompressobj(-zlib.MAX_WBITS).decompress(req_raw)
-        else:
-            req_unzip = req_raw
-        
-        req_dict = self.billing_req_to_dict(req_unzip)
-        request_ip = Utils.get_ip_addr(request)
-        
-        if req_dict is None:
-            self.logger.error(f"Failed to parse request {request.content.getvalue()}")
-            return b""
-
-        self.logger.debug(f"request {req_dict}")
-
-        rsa = RSA.import_key(open(self.config.billing.signing_key, "rb").read())
-        signer = PKCS1_v1_5.new(rsa)
-        digest = SHA.new()
-        traces: List[TraceData] = []
-
-        try:
-            for x in range(len(req_dict)):
-                if not req_dict[x]:
-                    continue
-                
-                if x == 0:                    
-                    req = BillingInfo(req_dict[x])
-                    continue
-
-                tmp = TraceData(req_dict[x])
-                if tmp.trace_type == TraceDataType.CHARGE:
-                    tmp = TraceDataCharge(req_dict[x])
-                elif tmp.trace_type == TraceDataType.EVENT:
-                    tmp = TraceDataEvent(req_dict[x])
-                elif tmp.trace_type == TraceDataType.CREDIT:
-                    tmp = TraceDataCredit(req_dict[x])
-                
-                traces.append(tmp)
-
-            kc_serial_bytes = req.keychipid.encode()
-        
-        except KeyError as e:
-            self.logger.error(f"Billing request failed to parse: {e}")
-            return f"result=5&linelimit=&message=field is missing or formatting is incorrect\r\n".encode()
-
-        machine = self.data.arcade.get_machine(req.keychipid)
-        if machine is None and not self.config.server.allow_unregistered_serials:
-            msg = f"Unrecognised serial {req.keychipid} attempted billing checkin from {request_ip} for {req.gameid} v{req.gamever}."
-            self.data.base.log_event(
-                "allnet", "BILLING_CHECKIN_NG_SERIAL", logging.WARN, msg
-            )
-            self.logger.warning(msg)
-
-            return f"result=1&requestno={req.requestno}&message=Keychip Serial bad\r\n".encode()
-
-        msg = (
-            f"Billing checkin from {request_ip}: game {req.gameid} ver {req.gamever} keychip {req.keychipid} playcount "
-            f"{req.playcnt} billing_type {req.billingtype.name} nearfull {req.nearfull} playlimit {req.playlimit}"
-        )
-        self.logger.info(msg)
-        self.data.base.log_event("billing", "BILLING_CHECKIN_OK", logging.INFO, msg)
-        if req.traceleft > 0:
-            self.logger.warn(f"{req.traceleft} unsent tracelogs")
-        kc_playlimit = req.playlimit
-        kc_nearfull = req.nearfull
-
-        while req.playcnt > req.playlimit:
-            kc_playlimit += 1024
-            kc_nearfull += 1024
-
-        playlimit = kc_playlimit
-        nearfull = kc_nearfull + (req.billingtype.value * 0x00010000)
-
-        digest.update(playlimit.to_bytes(4, "little") + kc_serial_bytes)
-        playlimit_sig = signer.sign(digest).hex()
-
-        digest = SHA.new()
-        digest.update(nearfull.to_bytes(4, "little") + kc_serial_bytes)
-        nearfull_sig = signer.sign(digest).hex()
-
-        # TODO: playhistory
-
-        #resp = BillingResponse(playlimit, playlimit_sig, nearfull, nearfull_sig)
-        resp = BillingResponse(playlimit, playlimit_sig, nearfull, nearfull_sig, req.requestno, req.protocolver)
-
-        resp_str = urllib.parse.unquote(urllib.parse.urlencode(vars(resp))) + "\r\n"
-
-        self.logger.debug(f"response {vars(resp)}")
-        if req.traceleft > 0:
-            self.logger.info(f"Requesting 20 more of {req.traceleft} unsent tracelogs")
-            return f"result=6&waittime=0&linelimit=20\r\n".encode()
-
-        return resp_str.encode("utf-8")
-
-    def handle_naomitest(self, request: Request, _: Dict) -> bytes:
+    async def handle_naomitest(self, request: Request) -> bytes:
         self.logger.info(f"Ping from {Utils.get_ip_addr(request)}")
-        return b"naomi ok"
-
-    def billing_req_to_dict(self, data: bytes):
-        """
-        Parses an billing request string into a python dictionary
-        """
-        try:
-            sections = data.decode("ascii").split("\r\n")
-
-            ret = []
-            for x in sections:
-                ret.append(dict(urllib.parse.parse_qsl(x)))
-            return ret
-
-        except Exception as e:
-            self.logger.error(f"billing_req_to_dict: {e} while parsing {data}")
-            return None
+        return PlainTextResponse("naomi ok")
 
     def allnet_req_to_dict(self, data: str) -> Optional[List[Dict[str, Any]]]:
         """
@@ -504,6 +449,150 @@ class AllnetServlet:
         zipped = zlib.compress(unzipped)
         return base64.b64encode(zipped)
 
+class BillingServlet:
+    def __init__(self, core_cfg: CoreConfig, cfg_folder: str) -> None:
+        self.config = core_cfg
+        self.config_folder = cfg_folder
+        self.data = Data(core_cfg)
+
+        self.logger = logging.getLogger("billing")
+        if not hasattr(self.logger, "initialized"):
+            log_fmt_str = "[%(asctime)s] Billing | %(levelname)s | %(message)s"
+            log_fmt = logging.Formatter(log_fmt_str)
+
+            fileHandler = TimedRotatingFileHandler(
+                "{0}/{1}.log".format(self.config.server.log_dir, "billing"),
+                when="d",
+                backupCount=10,
+            )
+            fileHandler.setFormatter(log_fmt)
+
+            consoleHandler = logging.StreamHandler()
+            consoleHandler.setFormatter(log_fmt)
+
+            self.logger.addHandler(fileHandler)
+            self.logger.addHandler(consoleHandler)
+
+            self.logger.setLevel(core_cfg.allnet.loglevel)
+            coloredlogs.install(
+                level=core_cfg.billing.loglevel, logger=self.logger, fmt=log_fmt_str
+            )
+            self.logger.initialized = True
+    
+    def startup(self) -> None:
+        self.logger.info(f"Ready on port {self.config.billing.port if self.config.billing.standalone else self.config.server.port}")
+
+    def billing_req_to_dict(self, data: bytes):
+        """
+        Parses an billing request string into a python dictionary
+        """
+        try:
+            sections = data.decode("ascii").split("\r\n")
+
+            ret = []
+            for x in sections:
+                ret.append(dict(urllib.parse.parse_qsl(x)))
+            return ret
+
+        except Exception as e:
+            self.logger.error(f"billing_req_to_dict: {e} while parsing {data}")
+            return None
+    
+    async def handle_billing_request(self, request: Request):
+        req_raw = await request.body()
+        
+        if request.headers.get('Content-Type', '') == "application/octet-stream":
+            req_unzip = zlib.decompressobj(-zlib.MAX_WBITS).decompress(req_raw)
+        else:
+            req_unzip = req_raw
+        
+        req_dict = self.billing_req_to_dict(req_unzip)
+        request_ip = Utils.get_ip_addr(request)
+        
+        if req_dict is None:
+            self.logger.error(f"Failed to parse request {req_raw}")
+            return PlainTextResponse()
+
+        self.logger.debug(f"request {req_dict}")
+
+        rsa = RSA.import_key(open(self.config.billing.signing_key, "rb").read())
+        signer = PKCS1_v1_5.new(rsa)
+        digest = SHA.new()
+        traces: List[TraceData] = []
+        try:
+            req = BillingInfo(req_dict[0])
+        except KeyError as e:
+            self.logger.error(f"Billing request failed to parse: {e}")
+            return PlainTextResponse("result=5&linelimit=&message=field is missing or formatting is incorrect\r\n")
+
+        for x in range(1, len(req_dict)):
+            if not req_dict[x]:
+                continue
+            
+            try:
+                tmp = TraceData(req_dict[x])
+                if tmp.trace_type == TraceDataType.CHARGE:
+                    tmp = TraceDataCharge(req_dict[x])
+                elif tmp.trace_type == TraceDataType.EVENT:
+                    tmp = TraceDataEvent(req_dict[x])
+                elif tmp.trace_type == TraceDataType.CREDIT:
+                    tmp = TraceDataCredit(req_dict[x])
+                
+                traces.append(tmp)
+            
+            except KeyError as e:
+                self.logger.warn(f"Tracelog failed to parse: {e}")
+
+        kc_serial_bytes = req.keychipid.encode()
+        
+
+        machine = await self.data.arcade.get_machine(req.keychipid)
+        if machine is None and not self.config.server.allow_unregistered_serials:
+            msg = f"Unrecognised serial {req.keychipid} attempted billing checkin from {request_ip} for {req.gameid} v{req.gamever}."
+            await self.data.base.log_event(
+                "allnet", "BILLING_CHECKIN_NG_SERIAL", logging.WARN, msg
+            )
+            self.logger.warning(msg)
+
+            return PlainTextResponse(f"result=1&requestno={req.requestno}&message=Keychip Serial bad\r\n")
+
+        msg = (
+            f"Billing checkin from {request_ip}: game {req.gameid} ver {req.gamever} keychip {req.keychipid} playcount "
+            f"{req.playcnt} billing_type {req.billingtype.name} nearfull {req.nearfull} playlimit {req.playlimit}"
+        )
+        self.logger.info(msg)
+        await self.data.base.log_event("billing", "BILLING_CHECKIN_OK", logging.INFO, msg)
+        if req.traceleft > 0:
+            self.logger.warn(f"{req.traceleft} unsent tracelogs")
+        kc_playlimit = req.playlimit
+        kc_nearfull = req.nearfull
+
+        while req.playcnt > req.playlimit:
+            kc_playlimit += 1024
+            kc_nearfull += 1024
+
+        playlimit = kc_playlimit
+        nearfull = kc_nearfull + (req.billingtype.value * 0x00010000)
+
+        digest.update(playlimit.to_bytes(4, "little") + kc_serial_bytes)
+        playlimit_sig = signer.sign(digest).hex()
+
+        digest = SHA.new()
+        digest.update(nearfull.to_bytes(4, "little") + kc_serial_bytes)
+        nearfull_sig = signer.sign(digest).hex()
+
+        # TODO: playhistory
+
+        resp = BillingResponse(playlimit, playlimit_sig, nearfull, nearfull_sig, req.requestno, req.protocolver)
+
+        resp_str = urllib.parse.unquote(urllib.parse.urlencode(vars(resp))) + "\r\n"
+
+        self.logger.debug(f"response {vars(resp)}")
+        if req.traceleft > 0:
+            self.logger.info(f"Requesting 20 more of {req.traceleft} unsent tracelogs")
+            return PlainTextResponse("result=6&waittime=0&linelimit=20\r\n")
+        
+        return PlainTextResponse(resp_str)
 
 class AllnetPowerOnRequest:
     def __init__(self, req: Dict) -> None:
@@ -525,7 +614,7 @@ class AllnetPowerOnResponse:
         self.stat = 1
         self.uri = ""
         self.host = ""
-        self.place_id = "123"
+        self.place_id = "0123"
         self.name = "ARTEMiS"
         self.nickname = "ARTEMiS"
         self.region0 = "1"
@@ -563,7 +652,6 @@ class AllnetPowerOnResponse3(AllnetPowerOnResponse):
         self.minute = None
         self.second = None
 
-
 class AllnetPowerOnResponse2(AllnetPowerOnResponse):
     def __init__(self) -> None:
         super().__init__()
@@ -573,7 +661,6 @@ class AllnetPowerOnResponse2(AllnetPowerOnResponse):
         self.timezone = "+09:00"
         self.res_class = "PowerOnResponseV2"
 
-
 class AllnetDownloadOrderRequest:
     def __init__(self, req: Dict) -> None:
         self.game_id = req.get("game_id", "")
@@ -581,7 +668,6 @@ class AllnetDownloadOrderRequest:
         self.serial = req.get("serial", "")
         self.encode = req.get("encode", "")
 
-
 class AllnetDownloadOrderResponse:
     def __init__(self, stat: int = 1, serial: str = "", uri: str = "") -> None:
         self.stat = stat
@@ -644,9 +730,17 @@ class TraceData:
             self.seq_number = int(data.get("sn", None))
             self.trace_type = TraceDataType(int(data.get("tt", None)))
             self.date_sync_flg = bool(data.get("ds", None))
+            
+            dt = data.get("dt", None)
+            if dt is None:
+                raise KeyError("dt not present")
+            if dt == "20000000000000": # Not sure what causes it to send like this...
+                self.date = datetime(2000, 1, 1, 0, 0, 0, 0)
+            else:
                 self.date = datetime.strptime(data.get("dt", None), BILLING_DT_FORMAT)
+            
             self.keychip = str(data.get("kn", None))
-            self.lib_ver = float(data.get("alib", None))
+            self.lib_ver = float(data.get("alib", 0))
         except Exception as e:
             raise KeyError(e)
 
@@ -654,8 +748,8 @@ class TraceDataCharge(TraceData):
     def __init__(self, data: Dict) -> None:
         super().__init__(data)
         try:
-            self.game_id = str(data.get("gi", None))
-            self.game_version = float(data.get("gv", None))
+            self.game_id = str(data.get("gi", None)) # these seem optional...?
+            self.game_version = float(data.get("gv", 0))
             self.board_serial = str(data.get("bn", None))
             self.shop_ip = str(data.get("ti", None))
             self.play_count = int(data.get("pc", None))
@@ -723,7 +817,6 @@ class BillingResponse:
         # playhistory -> YYYYMM/C:...
         # YYYY -> 4 digit year, MM -> 2 digit month, C -> Playcount during that period
 
-
 class AllnetRequestException(Exception):
     def __init__(self, message="") -> None:
         self.message = message
@@ -740,13 +833,13 @@ class DLReport:
         self.ot = data.get("ot")
         self.rt = data.get("rt")
         self.as_ = data.get("as")
-        self.rf_state = data.get("rf_state")
+        self.rf_state = DLI_STATUS.from_int(data.get("rf_state"))
         self.gd = data.get("gd")
         self.dav = data.get("dav")
         self.wdav = data.get("wdav") # app only
         self.dov = data.get("dov")
         self.wdov = data.get("wdov") # app only
-        self.__type = report_type
+        self.rep_type = report_type
         self.err = ""
     
     def validate(self) -> bool:
@@ -754,14 +847,6 @@ class DLReport:
             self.err = "serial not provided"
             return False
         
-        if self.dfl is None: 
-            self.err = "dfl not provided"
-            return False
-        
-        if self.wfl is None:
-            self.err = "wfl not provided"
-            return False
-        
         if self.tsc is None:
             self.err = "tsc not provided"
             return False
@@ -770,18 +855,6 @@ class DLReport:
             self.err = "tdsc not provided"
             return False
         
-        if self.at is None:
-            self.err = "at not provided"
-            return False
-        
-        if self.ot is None:
-            self.err = "ot not provided"
-            return False
-        
-        if self.rt is None:
-            self.err = "rt not provided"
-            return False
-        
         if self.as_ is None:
             self.err = "as not provided"
             return False
@@ -802,12 +875,57 @@ class DLReport:
             self.err = "dov not provided"
             return False
         
-        if (self.wdav is None or self.wdov is None) and self.__type == DLIMG_TYPE.app:
+        if (self.wdav is None or self.wdov is None) and self.rep_type == DLIMG_TYPE.app:
             self.err = "wdav or wdov not provided in app image"
             return False
         
-        if (self.wdav is not None or self.wdov is not None) and self.__type == DLIMG_TYPE.opt:
+        if (self.wdav is not None or self.wdov is not None) and self.rep_type == DLIMG_TYPE.opt:
             self.err = "wdav or wdov provided in opt image"
             return False
         
         return True
+
+cfg_dir = environ.get("DIANA_CFG_DIR", "config")
+cfg: CoreConfig = CoreConfig()
+if path.exists(f"{cfg_dir}/core.yaml"):
+    cfg.update(yaml.safe_load(open(f"{cfg_dir}/core.yaml")))
+
+if not path.exists(cfg.server.log_dir):
+    mkdir(cfg.server.log_dir)
+
+if not access(cfg.server.log_dir, W_OK):
+    print(
+        f"Log directory {cfg.server.log_dir} NOT writable, please check permissions"
+    )
+    exit(1)
+
+billing = BillingServlet(cfg, cfg_dir)
+app_billing = Starlette(
+    cfg.server.is_develop, 
+    [        
+        Route("/request", billing.handle_billing_request, methods=["POST"]),
+        Route("/request/", billing.handle_billing_request, methods=["POST"]),
+    ],
+    on_startup=[billing.startup]
+)
+
+allnet = AllnetServlet(cfg, cfg_dir)
+route_lst = [
+    Route("/sys/servlet/PowerOn", allnet.handle_poweron, methods=["GET", "POST"]),
+    Route("/sys/servlet/DownloadOrder", allnet.handle_dlorder, methods=["GET", "POST"]),
+    Route("/sys/servlet/LoaderStateRecorder", allnet.handle_loaderstaterecorder, methods=["GET", "POST"]),
+    Route("/sys/servlet/Alive", allnet.handle_alive, methods=["GET", "POST"]),
+    Route("/naomitest.html", allnet.handle_naomitest),
+]
+
+if cfg.allnet.allow_online_updates:
+    route_lst += [
+        Route("/report-api/Report", allnet.handle_dlorder_report, methods=["POST"]),
+        Route("/dl/ini/{file:str}", allnet.handle_dlorder_ini),
+    ]
+
+app_allnet = Starlette(
+    cfg.server.is_develop, 
+    route_lst,
+    on_startup=[allnet.startup]
+)

core/app.py

@@ -0,0 +1,92 @@
+import yaml
+import logging
+import coloredlogs
+from logging.handlers import TimedRotatingFileHandler
+from starlette.routing import Route
+from starlette.requests import Request
+from starlette.applications import Starlette
+from starlette.responses import PlainTextResponse
+from os import environ, path, mkdir, W_OK, access
+from typing import List
+
+from core import CoreConfig, TitleServlet, MuchaServlet, AllnetServlet, BillingServlet, AimedbServlette
+from core.frontend import FrontendServlet
+
+async def dummy_rt(request: Request):
+    return PlainTextResponse("Service OK")
+
+cfg_dir = environ.get("ARTEMIS_CFG_DIR", "config")
+cfg: CoreConfig = CoreConfig()
+if path.exists(f"{cfg_dir}/core.yaml"):
+    cfg.update(yaml.safe_load(open(f"{cfg_dir}/core.yaml")))
+
+if not path.exists(cfg.server.log_dir):
+    mkdir(cfg.server.log_dir)
+
+if not access(cfg.server.log_dir, W_OK):
+    print(
+        f"Log directory {cfg.server.log_dir} NOT writable, please check permissions"
+    )
+    exit(1)
+
+logger = logging.getLogger("core")
+log_fmt_str = "[%(asctime)s] Core | %(levelname)s | %(message)s"
+log_fmt = logging.Formatter(log_fmt_str)
+
+fileHandler = TimedRotatingFileHandler(
+    "{0}/{1}.log".format(cfg.server.log_dir, "core"), when="d", backupCount=10
+)
+fileHandler.setFormatter(log_fmt)
+
+consoleHandler = logging.StreamHandler()
+consoleHandler.setFormatter(log_fmt)
+
+logger.addHandler(fileHandler)
+logger.addHandler(consoleHandler)
+
+log_lv = logging.DEBUG if cfg.server.is_develop else logging.INFO
+logger.setLevel(log_lv)
+coloredlogs.install(level=log_lv, logger=logger, fmt=log_fmt_str)
+
+logger.info(f"Artemis starting in {'develop' if cfg.server.is_develop else 'production'} mode")
+
+title = TitleServlet(cfg, cfg_dir) # This has to be loaded first to load plugins
+mucha = MuchaServlet(cfg, cfg_dir)
+
+route_lst: List[Route] = [
+    # Mucha
+    Route("/mucha_front/boardauth.do", mucha.handle_boardauth, methods=["POST"]),
+    Route("/mucha_front/updatacheck.do", mucha.handle_updatecheck, methods=["POST"]),
+    Route("/mucha_front/downloadstate.do", mucha.handle_dlstate, methods=["POST"]),
+    # General
+    Route("/", dummy_rt),
+    Route("/robots.txt", FrontendServlet.robots)
+]
+
+if not cfg.billing.standalone:
+    billing = BillingServlet(cfg, cfg_dir)
+    route_lst += [
+        Route("/request", billing.handle_billing_request, methods=["POST"]),
+        Route("/request/", billing.handle_billing_request, methods=["POST"]),
+    ]
+
+if not cfg.allnet.standalone:
+    allnet = AllnetServlet(cfg, cfg_dir)
+    route_lst += [
+        Route("/sys/servlet/PowerOn", allnet.handle_poweron, methods=["GET", "POST"]),
+        Route("/sys/servlet/DownloadOrder", allnet.handle_dlorder, methods=["GET", "POST"]),
+        Route("/sys/servlet/LoaderStateRecorder", allnet.handle_loaderstaterecorder, methods=["GET", "POST"]),
+        Route("/sys/servlet/Alive", allnet.handle_alive, methods=["GET", "POST"]),
+        Route("/naomitest.html", allnet.handle_naomitest),
+    ]
+    
+    if cfg.allnet.allow_online_updates:
+        route_lst += [
+            Route("/report-api/Report", allnet.handle_dlorder_report, methods=["POST"]),
+            Route("/dl/ini/{file:str}", allnet.handle_dlorder_ini),
+        ]
+
+for code, game in title.title_registry.items():
+    route_lst += game.get_routes()
+
+app = Starlette(cfg.server.is_develop, route_lst)

core/config.py

@@ -1,17 +1,49 @@
 import logging, os
 from typing import Any
 
-
 class ServerConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
     @property
     def listen_address(self) -> str:
+        """
+        Address Artemis will bind to and listen on
+        """
         return CoreConfig.get_config_field(
             self.__config, "core", "server", "listen_address", default="127.0.0.1"
         )
     
+    @property
+    def hostname(self) -> str:
+        """
+        Hostname sent to games
+        """
+        return CoreConfig.get_config_field(
+            self.__config, "core", "server", "hostname", default="localhost"
+        )
+    
+    @property
+    def port(self) -> int:
+        """
+        Port the game will listen on
+        """
+        return CoreConfig.get_config_field(
+            self.__config, "core", "server", "port", default=80
+        )
+
+    @property
+    def ssl_key(self) -> str:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "server", "ssl_key", default="cert/title.key"
+        )
+
+    @property
+    def ssl_cert(self) -> str:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "title", "ssl_cert", default="cert/title.pem"
+        )
+
     @property
     def allow_user_registration(self) -> bool:
         return CoreConfig.get_config_field(
@@ -37,9 +69,29 @@ class ServerConfig:
         )
 
     @property
-    def threading(self) -> bool:
+    def is_using_proxy(self) -> bool:
         return CoreConfig.get_config_field(
-            self.__config, "core", "server", "threading", default=False
+            self.__config, "core", "server", "is_using_proxy", default=False
+        )
+
+    @property
+    def proxy_port(self) -> int:
+        """
+        What port the proxy is listening on. This will be sent instead of 'port' if 
+        is_using_proxy is True and this value is non-zero
+        """
+        return CoreConfig.get_config_field(
+            self.__config, "core", "server", "proxy_port", default=0
+        )
+
+    @property
+    def proxy_port_ssl(self) -> int:
+        """
+        What port the proxy is listening for secure connections on. This will be sent 
+        instead of 'port' if is_using_proxy is True and this value is non-zero
+        """
+        return CoreConfig.get_config_field(
+            self.__config, "core", "server", "proxy_port_ssl", default=0
         )
 
     @property
@@ -60,7 +112,6 @@ class ServerConfig:
             self.__config, "core", "server", "strict_ip_checking", default=False
         )
 
-
 class TitleConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
@@ -74,18 +125,17 @@ class TitleConfig:
         )
 
     @property
-    def hostname(self) -> str:
+    def reboot_start_time(self) -> str:
         return CoreConfig.get_config_field(
-            self.__config, "core", "title", "hostname", default="localhost"
+            self.__config, "core", "title", "reboot_start_time", default=""
         )
 
     @property
-    def port(self) -> int:
+    def reboot_end_time(self) -> str:
         return CoreConfig.get_config_field(
-            self.__config, "core", "title", "port", default=8080
+            self.__config, "core", "title", "reboot_end_time", default=""
         )
 
-
 class DatabaseConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
@@ -123,7 +173,7 @@ class DatabaseConfig:
     @property
     def protocol(self) -> str:
         return CoreConfig.get_config_field(
-            self.__config, "core", "database", "type", default="mysql"
+            self.__config, "core", "database", "protocol", default="mysql"
         )
 
     @property
@@ -141,13 +191,9 @@ class DatabaseConfig:
         )
 
     @property
-    def user_table_autoincrement_start(self) -> int:
+    def enable_memcached(self) -> bool:
         return CoreConfig.get_config_field(
-            self.__config,
-            "core",
-            "database",
-            "user_table_autoincrement_start",
-            default=10000,
+            self.__config, "core", "database", "enable_memcached", default=True
         )
 
     @property
@@ -156,13 +202,12 @@ class DatabaseConfig:
             self.__config, "core", "database", "memcached_host", default="localhost"
         )
 
-
 class FrontendConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
     @property
-    def enable(self) -> int:
+    def enable(self) -> bool:
         return CoreConfig.get_config_field(
             self.__config, "core", "frontend", "enable", default=False
         )
@@ -170,7 +215,7 @@ class FrontendConfig:
     @property
     def port(self) -> int:
         return CoreConfig.get_config_field(
-            self.__config, "core", "frontend", "port", default=8090
+            self.__config, "core", "frontend", "port", default=8080
         )
 
     @property
@@ -181,17 +226,20 @@ class FrontendConfig:
             )
         )
     
+    @property
+    def secret(self) -> str:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "frontend", "secret", default=""
+        )
 
 class AllnetConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
     @property
-    def loglevel(self) -> int:
-        return CoreConfig.str_to_loglevel(
-            CoreConfig.get_config_field(
-                self.__config, "core", "allnet", "loglevel", default="info"
-            )
+    def standalone(self) -> bool:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "allnet", "standalone", default=False
         )
     
     @property
@@ -201,9 +249,11 @@ class AllnetConfig:
         )
 
     @property
-    def ip_check(self) -> bool:
-        return CoreConfig.get_config_field(
-            self.__config, "core", "allnet", "ip_check", default=False
+    def loglevel(self) -> int:
+        return CoreConfig.str_to_loglevel(
+            CoreConfig.get_config_field(
+                self.__config, "core", "allnet", "loglevel", default="info"
+            )
         )
 
     @property
@@ -218,11 +268,24 @@ class AllnetConfig:
             self.__config, "core", "allnet", "update_cfg_folder", default=""
         )
 
-
 class BillingConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
     
+    @property
+    def standalone(self) -> bool:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "billing", "standalone", default=True
+        )
+
+    @property
+    def loglevel(self) -> int:
+        return CoreConfig.str_to_loglevel(
+            CoreConfig.get_config_field(
+                self.__config, "core", "billing", "loglevel", default="info"
+            )
+        )
+
     @property
     def port(self) -> int:
         return CoreConfig.get_config_field(
@@ -247,11 +310,22 @@ class BillingConfig:
             self.__config, "core", "billing", "signing_key", default="cert/billing.key"
         )
 
-
 class AimedbConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
+    @property
+    def enable(self) -> bool:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "aimedb", "enable", default=True
+        )
+
+    @property
+    def listen_address(self) -> bool:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "aimedb", "listen_address", default=""
+        )
+
     @property
     def loglevel(self) -> int:
         return CoreConfig.str_to_loglevel(
@@ -272,17 +346,22 @@ class AimedbConfig:
             self.__config, "core", "aimedb", "key", default=""
         )
 
+    @property
+    def id_secret(self) -> str:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "aimedb", "id_secret", default=""
+        )
+
+    @property
+    def id_lifetime_seconds(self) -> int:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "aimedb", "id_lifetime_seconds", default=86400
+        )
 
 class MuchaConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
-    @property
-    def enable(self) -> int:
-        return CoreConfig.get_config_field(
-            self.__config, "core", "mucha", "enable", default=False
-        )
-
     @property
     def loglevel(self) -> int:
         return CoreConfig.str_to_loglevel(
@@ -291,13 +370,6 @@ class MuchaConfig:
             )
         )
 
-    @property
-    def hostname(self) -> str:
-        return CoreConfig.get_config_field(
-            self.__config, "core", "mucha", "hostname", default="localhost"
-        )
-
-
 class CoreConfig(dict):
     def __init__(self) -> None:
         self.server = ServerConfig(self)
@@ -320,6 +392,19 @@ class CoreConfig(dict):
         else:
             return logging.INFO
     
+    @classmethod
+    def loglevel_to_str(cls, level: int) -> str:
+        if level == logging.ERROR:
+            return "error"
+        elif level == logging.WARN:
+            return "warn"
+        elif level == logging.INFO:
+            return "info"
+        elif level == logging.DEBUG:
+            return "debug"
+        else:
+            return "notset"
+
     @classmethod
     def get_config_field(
         cls, __config: dict, module, *path: str, default: Any = ""

core/data/alembic/README

@@ -0,0 +1 @@
+Generic single-database configuration.

core/data/alembic/alembic.ini

@@ -0,0 +1,64 @@
+# A generic, single database configuration.
+
+[alembic]
+script_location=.
+
+# template used to generate migration files
+# file_template = %%(rev)s_%%(slug)s
+
+# max length of characters to apply to the
+# "slug" field
+#truncate_slug_length = 40
+
+# set to 'true' to run the environment during
+# the 'revision' command, regardless of autogenerate
+# revision_environment = false
+
+# set to 'true' to allow .pyc and .pyo files without
+# a source .py file to be detected as revisions in the
+# versions/ directory
+# sourceless = false
+
+# version location specification; this defaults
+# to migrations//versions.  When using multiple version
+# directories, initial revisions must be specified with --version-path
+# version_locations = %(here)s/bar %(here)s/bat migrations//versions
+
+# the output encoding used when revision files
+# are written from script.py.mako
+# output_encoding = utf-8
+
+# Logging configuration
+[loggers]
+keys = root,sqlalchemy,alembic
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARN
+handlers = console
+qualname =
+
+[logger_sqlalchemy]
+level = WARN
+handlers =
+qualname = sqlalchemy.engine
+
+[logger_alembic]
+level = INFO
+handlers =
+qualname = alembic
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S

core/data/alembic/env.py

@@ -0,0 +1,81 @@
+from __future__ import with_statement
+from alembic import context
+from sqlalchemy import engine_from_config, pool
+from logging.config import fileConfig
+
+from core.data.schema.base import metadata
+
+# this is the Alembic Config object, which provides
+# access to the values within the .ini file in use.
+config = context.config
+
+# Interpret the config file for Python logging.
+# This line sets up loggers basically.
+fileConfig(config.config_file_name)
+
+# add your model's MetaData object here
+# for 'autogenerate' support
+# from myapp import mymodel
+# target_metadata = mymodel.Base.metadata
+target_metadata = metadata
+
+# other values from the config, defined by the needs of env.py,
+# can be acquired:
+# my_important_option = config.get_main_option("my_important_option")
+# ... etc.
+
+
+def run_migrations_offline():
+    """Run migrations in 'offline' mode.
+
+    This configures the context with just a URL
+    and not an Engine, though an Engine is acceptable
+    here as well.  By skipping the Engine creation
+    we don't even need a DBAPI to be available.
+
+    Calls to context.execute() here emit the given string to the
+    script output.
+
+    """
+    raise Exception('Not implemented or configured!')
+
+    url = config.get_main_option("sqlalchemy.url")
+    context.configure(
+        url=url, target_metadata=target_metadata, literal_binds=True)
+
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+def run_migrations_online():
+    """Run migrations in 'online' mode.
+
+    In this scenario we need to create an Engine
+    and associate a connection with the context.
+
+    """
+    ini_section = config.get_section(config.config_ini_section)
+    overrides = context.get_x_argument(as_dictionary=True)
+    for override in overrides:
+        ini_section[override] = overrides[override]
+
+    connectable = engine_from_config(
+        ini_section,
+        prefix='sqlalchemy.',
+        poolclass=pool.NullPool)
+
+    with connectable.connect() as connection:
+        context.configure(
+            connection=connection,
+            target_metadata=target_metadata,
+            compare_type=True,
+            compare_server_default=True,
+        )
+
+        with context.begin_transaction():
+            context.run_migrations()
+
+if context.is_offline_mode():
+    run_migrations_offline()
+else:
+    run_migrations_online()

core/data/alembic/script.py.mako

@@ -0,0 +1,24 @@
+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision | comma,n}
+Create Date: ${create_date}
+
+"""
+from alembic import op
+import sqlalchemy as sa
+${imports if imports else ""}
+
+# revision identifiers, used by Alembic.
+revision = ${repr(up_revision)}
+down_revision = ${repr(down_revision)}
+branch_labels = ${repr(branch_labels)}
+depends_on = ${repr(depends_on)}
+
+
+def upgrade():
+    ${upgrades if upgrades else "pass"}
+
+
+def downgrade():
+    ${downgrades if downgrades else "pass"}

core/data/alembic/versions/835b862f9bf0_initial_migration.py

@@ -0,0 +1,24 @@
+"""Initial Migration
+
+Revision ID: 835b862f9bf0
+Revises: 
+Create Date: 2024-01-09 13:06:10.787432
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = '835b862f9bf0'
+down_revision = None
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    pass
+
+
+def downgrade():
+    pass

core/data/alembic/versions/d8950c7ce2fc_remove_old_db_mgmt_system.py

@@ -0,0 +1,29 @@
+"""Remove old db mgmt system
+
+Revision ID: d8950c7ce2fc
+Revises: 835b862f9bf0
+Create Date: 2024-01-09 13:43:51.381175
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = 'd8950c7ce2fc'
+down_revision = '835b862f9bf0'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    op.drop_table("schema_versions")
+
+
+def downgrade():
+    op.create_table(
+        "schema_versions",
+        sa.Column("game", sa.String(4), primary_key=True, nullable=False),
+        sa.Column("version", sa.Integer, nullable=False, server_default="1"),
+        mysql_charset="utf8mb4",
+    )

core/data/cache.py

@@ -17,7 +17,7 @@ except ModuleNotFoundError:
 
 def cached(lifetime: int = 10, extra_key: Any = None) -> Callable:
     def _cached(func: Callable) -> Callable:
-        if has_mc:
+        if has_mc and (cfg and cfg.database.enable_memcached):
             hostname = "127.0.0.1"
             if cfg:
                 hostname = cfg.database.memcached_host

core/data/database.py

@@ -1,13 +1,14 @@
 import logging, coloredlogs
-from typing import Optional, Dict, List
+from typing import Optional
 from sqlalchemy.orm import scoped_session, sessionmaker
-from sqlalchemy.exc import SQLAlchemyError
 from sqlalchemy import create_engine
 from logging.handlers import TimedRotatingFileHandler
-import importlib, os
+import os
 import secrets, string
 import bcrypt
 from hashlib import sha256
+import alembic.config
+import glob
 
 from core.config import CoreConfig
 from core.data.schema import *
@@ -15,7 +16,6 @@ from core.utils import Utils
 
 
 class Data:
-    current_schema_version = 6
     engine = None
     session = None
     user = None
@@ -77,281 +77,173 @@ class Data:
             )
             self.logger.handler_set = True  # type: ignore
 
+    def __alembic_cmd(self, command: str, *args: str) -> None:
+        old_dir = os.path.abspath(os.path.curdir)
+        base_dir = os.path.join(os.path.abspath(os.path.curdir), 'core', 'data', 'alembic')
+        alembicArgs = [
+            "-c",
+            os.path.join(base_dir, "alembic.ini"),
+            "-x",
+            f"script_location={base_dir}",
+            "-x",
+            f"sqlalchemy.url={self.__url}",
+            command,
+        ]
+        alembicArgs.extend(args)
+        os.chdir(base_dir)
+        alembic.config.main(argv=alembicArgs)
+        os.chdir(old_dir)
+
     def create_database(self):
         self.logger.info("Creating databases...")
-        try:
-            metadata.create_all(self.__engine.connect())
-        except SQLAlchemyError as e:
-            self.logger.error(f"Failed to create databases! {e}")
-            return
-
-        games = Utils.get_all_titles()
-        for game_dir, game_mod in games.items():
-            try:
-                if hasattr(game_mod, "database") and hasattr(
-                    game_mod, "current_schema_version"
-                ):
-                    game_mod.database(self.config)
-                metadata.create_all(self.__engine.connect())
-
-                self.base.touch_schema_ver(
-                    game_mod.current_schema_version, game_mod.game_codes[0]
+        metadata.create_all(
+            self.engine,
+            checkfirst=True,
         )
 
-            except Exception as e:
-                self.logger.warning(
-                    f"Could not load database schema from {game_dir} - {e}"
-                )
-
-        self.logger.info(f"Setting base_schema_ver to {self.current_schema_version}")
-        self.base.set_schema_ver(self.current_schema_version)
-
-        self.logger.info(
-            f"Setting user auto_incrememnt to {self.config.database.user_table_autoincrement_start}"
-        )
-        self.user.reset_autoincrement(
-            self.config.database.user_table_autoincrement_start
-        )
-
-    def recreate_database(self):
-        self.logger.info("Dropping all databases...")
-        self.base.execute("SET FOREIGN_KEY_CHECKS=0")
-        try:
-            metadata.drop_all(self.__engine.connect())
-        except SQLAlchemyError as e:
-            self.logger.error(f"Failed to drop databases! {e}")
-            return
-
-        for root, dirs, files in os.walk("./titles"):
-            for dir in dirs:
-                if not dir.startswith("__"):
-                    try:
-                        mod = importlib.import_module(f"titles.{dir}")
-
-                        try:
+        for _, mod in Utils.get_all_titles().items():
             if hasattr(mod, "database"):
                 mod.database(self.config)
-                            metadata.drop_all(self.__engine.connect())
-
-                        except Exception as e:
-                            self.logger.warning(
-                                f"Could not load database schema from {dir} - {e}"
+                metadata.create_all(
+                    self.engine,
+                    checkfirst=True,
                 )
 
-                    except ImportError as e:
-                        self.logger.warning(
-                            f"Failed to load database schema dir {dir} - {e}"
-                        )
-            break
-
-        self.base.execute("SET FOREIGN_KEY_CHECKS=1")
-
-        self.create_database()
-
-    def migrate_database(self, game: str, version: Optional[int], action: str) -> None:
-        old_ver = self.base.get_schema_ver(game)
-        sql = ""
-        if version is None:
-            if not game == "CORE":
-                titles = Utils.get_all_titles()
-
-                for folder, mod in titles.items():
-                    if not mod.game_codes[0] == game:
-                        continue
-
-                    if hasattr(mod, "current_schema_version"):
-                        version = mod.current_schema_version
-
-                    else:
-                        self.logger.warning(
-                            f"current_schema_version not found for {folder}"
+        # Stamp the end revision as if alembic had created it, so it can take off after this.
+        self.__alembic_cmd(
+            "stamp",
+            "head",
         )
 
-            else:
-                version = self.current_schema_version
-
-        if version is None:
-            self.logger.warning(
-                f"Could not determine latest version for {game}, please specify --version"
+    def schema_upgrade(self, ver: str = None):
+        self.__alembic_cmd(
+            "upgrade",
+            "head" if not ver else ver,
         )
 
-        if old_ver is None:
-            self.logger.error(
-                f"Schema for game {game} does not exist, did you run the creation script?"
+    def schema_downgrade(self, ver: str):
+        self.__alembic_cmd(
+            "downgrade",
+            ver,
         )
-            return
 
-        if old_ver == version:
-            self.logger.info(
-                f"Schema for game {game} is already version {old_ver}, nothing to do"
-            )
-            return
-
-        if action == "upgrade":
-            for x in range(old_ver, version):
-                if not os.path.exists(
-                    f"core/data/schema/versions/{game.upper()}_{x + 1}_{action}.sql"
-                ):
-                    self.logger.error(
-                        f"Could not find {action} script {game.upper()}_{x + 1}_{action}.sql in core/data/schema/versions folder"
-                    )
-                    return
-
-                with open(
-                    f"core/data/schema/versions/{game.upper()}_{x + 1}_{action}.sql",
-                    "r",
-                    encoding="utf-8",
-                ) as f:
-                    sql = f.read()
-
-                result = self.base.execute(sql)
-                if result is None:
-                    self.logger.error("Error execuing sql script!")
-                    return None
-
-        else:
-            for x in range(old_ver, version, -1):
-                if not os.path.exists(
-                    f"core/data/schema/versions/{game.upper()}_{x - 1}_{action}.sql"
-                ):
-                    self.logger.error(
-                        f"Could not find {action} script {game.upper()}_{x - 1}_{action}.sql in core/data/schema/versions folder"
-                    )
-                    return
-
-                with open(
-                    f"core/data/schema/versions/{game.upper()}_{x - 1}_{action}.sql",
-                    "r",
-                    encoding="utf-8",
-                ) as f:
-                    sql = f.read()
-
-                result = self.base.execute(sql)
-                if result is None:
-                    self.logger.error("Error execuing sql script!")
-                    return None
-
-        result = self.base.set_schema_ver(version, game)
-        if result is None:
-            self.logger.error("Error setting version in schema_version table!")
-            return None
-
-        self.logger.info(f"Successfully migrated {game} to schema version {version}")
-
-    def create_owner(self, email: Optional[str] = None) -> None:
+    async def create_owner(self, email: Optional[str] = None, code: Optional[str] = "00000000000000000000") -> None:
         pw = "".join(
             secrets.choice(string.ascii_letters + string.digits) for i in range(20)
         )
         hash = bcrypt.hashpw(pw.encode(), bcrypt.gensalt())
 
-        user_id = self.user.create_user(email=email, permission=255, password=hash)
+        user_id = await self.user.create_user(username="sysowner", email=email, password=hash.decode(), permission=255)
         if user_id is None:
             self.logger.error(f"Failed to create owner with email {email}")
             return
 
-        card_id = self.card.create_card(user_id, "00000000000000000000")
+        card_id = await self.card.create_card(user_id, code)
         if card_id is None:
             self.logger.error(f"Failed to create card for owner with id {user_id}")
             return
 
         self.logger.warning(
-            f"Successfully created owner with email {email}, access code 00000000000000000000, and password {pw} Make sure to change this password and assign a real card ASAP!"
+            f"Successfully created owner with email {email}, access code {code}, and password {pw} Make sure to change this password and assign a real card ASAP!"
         )
     
-    def migrate_card(self, old_ac: str, new_ac: str, should_force: bool) -> None:
-        if old_ac == new_ac:
-            self.logger.error("Both access codes are the same!")
+    async def migrate(self) -> None:
+        exist = await self.base.execute("SELECT * FROM alembic_version")
+        if exist is not None:
+            self.logger.warn("No need to migrate as you have already migrated to alembic. If you are trying to upgrade the schema, use `upgrade` instead!")
             return
         
-        new_card = self.card.get_card_by_access_code(new_ac)
-        if new_card is None:
-            self.card.update_access_code(old_ac, new_ac)
+        self.logger.info("Upgrading to latest with legacy system")
+        if not await self.legacy_upgrade():
+            self.logger.warn("No need to migrate as you have already deleted the old schema_versions system. If you are trying to upgrade the schema, use `upgrade` instead!")
             return
+        self.logger.info("Done")
         
-        if not should_force:
-            self.logger.warning(
-                f"Card already exists for access code {new_ac} (id {new_card['id']}). If you wish to continue, rerun with the '--force' flag."
-                f" All exiting data on the target card {new_ac} will be perminently erased and replaced with data from card {old_ac}."
+        self.logger.info("Stamp with initial revision")
+        self.__alembic_cmd(
+            "stamp",
+            "835b862f9bf0",
         )
-            return
 
-        self.logger.info(
-            f"All exiting data on the target card {new_ac} will be perminently erased and replaced with data from card {old_ac}."
+        self.logger.info("Upgrade")
+        self.__alembic_cmd(
+            "upgrade",
+            "head",
         )
-        self.card.delete_card(new_card["id"])
-        self.card.update_access_code(old_ac, new_ac)
     
-        hanging_user = self.user.get_user(new_card["user"])
-        if hanging_user["password"] is None:
-            self.logger.info(f"Delete hanging user {hanging_user['id']}")
-            self.user.delete_user(hanging_user["id"])
+    async def legacy_upgrade(self) -> bool:
+        vers = await self.base.execute("SELECT * FROM schema_versions")
+        if vers is None:
+            self.logger.warn("Cannot legacy upgrade, schema_versions table unavailable!")
+            return False
         
-    def delete_hanging_users(self) -> None:
-        """
-        Finds and deletes users that have not registered for the webui that have no cards assocated with them.
-        """
-        unreg_users = self.user.get_unregistered_users()
-        if unreg_users is None:
-            self.logger.error("Error occoured finding unregistered users")
+        db_vers = {}
+        vers_list = vers.fetchall()
+        for x in vers_list:
+            db_vers[x['game']] = x['version']
         
-        for user in unreg_users:
-            cards = self.card.get_user_cards(user["id"])
-            if cards is None:
-                self.logger.error(f"Error getting cards for user {user['id']}")
-                continue
+        core_now_ver = int(db_vers['CORE']) + 1
+        while os.path.exists(f"core/data/schema/versions/CORE_{core_now_ver}_upgrade.sql"):
+            with open(f"core/data/schema/versions/CORE_{core_now_ver}_upgrade.sql", "r") as f:
+                result = await self.base.execute(f.read())
                 
-            if not cards:
-                self.logger.info(f"Delete hanging user {user['id']}")
-                self.user.delete_user(user["id"])
-
-    def autoupgrade(self) -> None:
-        all_game_versions = self.base.get_all_schema_vers()
-        if all_game_versions is None:
-            self.logger.warning("Failed to get schema versions")
-            return
-
-        all_games = Utils.get_all_titles()
-        all_games_list: Dict[str, int] = {}
-        for _, mod in all_games.items():
-            if hasattr(mod, "current_schema_version"):
-                all_games_list[mod.game_codes[0]] = mod.current_schema_version
-
-        for x in all_game_versions:
-            failed = False
-            game = x["game"].upper()
-            update_ver = int(x["version"])
-            latest_ver = all_games_list.get(game, 1)
-            if game == "CORE":
-                latest_ver = self.current_schema_version
-
-            if update_ver == latest_ver:
-                self.logger.info(f"{game} is already latest version")
-                continue
-
-            for y in range(update_ver + 1, latest_ver + 1):
-                if os.path.exists(f"core/data/schema/versions/{game}_{y}_upgrade.sql"):
-                    with open(
-                        f"core/data/schema/versions/{game}_{y}_upgrade.sql",
-                        "r",
-                        encoding="utf-8",
-                    ) as f:
-                        sql = f.read()
-
-                    result = self.base.execute(sql)
                 if result is None:
-                        self.logger.error(
-                            f"Error execuing sql script for game {game} v{y}!"
-                        )
-                        failed = True
+                    self.logger.error(f"Invalid upgrade script CORE_{core_now_ver}_upgrade.sql")
                     break
-                else:
-                    self.logger.warning(f"Could not find script {game}_{y}_upgrade.sql")
-                    failed = True
 
-            if not failed:
-                self.base.set_schema_ver(latest_ver, game)
+                result = await self.base.execute(f"UPDATE schema_versions SET version = {core_now_ver} WHERE game = 'CORE'")
+                if result is None:
+                    self.logger.error(f"Failed to update schema version for CORE to {core_now_ver}")
+                    break
             
-    def show_versions(self) -> None:
-        all_game_versions = self.base.get_all_schema_vers()
-        for ver in all_game_versions:
-            self.logger.info(f"{ver['game']} -> v{ver['version']}")
+            self.logger.info(f"Upgrade CORE to version {core_now_ver}")
+            core_now_ver += 1
+        
+        for _, mod in Utils.get_all_titles().items():
+            game_codes = getattr(mod, "game_codes", [])
+            for game in game_codes:
+                if game not in db_vers:
+                    self.logger.warn(f"{game} does not have an antry in schema_versions, skipping")
+                    continue
+
+                now_ver = int(db_vers[game]) + 1
+                while os.path.exists(f"core/data/schema/versions/{game}_{now_ver}_upgrade.sql"):
+                    with open(f"core/data/schema/versions/{game}_{now_ver}_upgrade.sql", "r") as f:
+                        result = await self.base.execute(f.read())
+                        
+                        if result is None:
+                            self.logger.error(f"Invalid upgrade script {game}_{now_ver}_upgrade.sql")
+                            break
+
+                        result = await self.base.execute(f"UPDATE schema_versions SET version = {now_ver} WHERE game = '{game}'")
+                        if result is None:
+                            self.logger.error(f"Failed to update schema version for {game} to {now_ver}")
+                            break
+
+                    self.logger.info(f"Upgrade {game} to version {now_ver}")
+                    now_ver += 1
+        
+        return True
+
+    async def create_revision(self, message: str) -> None:
+        if not message:
+            self.logger.info("Message is required for create-revision")
+            return
+        
+        self.__alembic_cmd(
+            "revision",
+            "-m",
+            message,
+        )
+
+    async def create_revision_auto(self, message: str) -> None:
+        if not message:
+            self.logger.info("Message is required for create-revision")
+            return
+        
+        self.__alembic_cmd(
+            "revision",
+            "--autogenerate",
+            "-m",
+            message,
+        )

core/data/schema/arcade.py

@@ -41,7 +41,6 @@ machine = Table(
     Column("country", String(3)),  # overwrites if not null
     Column("timezone", String(255)),
     Column("ota_enable", Boolean),
-    Column("is_cab", Boolean),
     Column("memo", String(255)),
     Column("is_cab", Boolean),
     Column("data", JSON),
@@ -70,7 +69,7 @@ arcade_owner = Table(
 
 
 class ArcadeData(BaseData):
-    def get_machine(self, serial: str = None, id: int = None) -> Optional[Row]:
+    async def get_machine(self, serial: str = None, id: int = None) -> Optional[Row]:
         if serial is not None:
             serial = serial.replace("-", "")
             if len(serial) == 11:
@@ -90,12 +89,12 @@ class ArcadeData(BaseData):
             self.logger.error(f"{__name__ }: Need either serial or ID to look up!")
             return None
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchone()
 
-    def put_machine(
+    async def put_machine(
         self,
         arcade_id: int,
         serial: str = "",
@@ -111,13 +110,13 @@ class ArcadeData(BaseData):
             arcade=arcade_id, keychip=serial, board=board, game=game, is_cab=is_cab
         )
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.lastrowid
 
-    def set_machine_serial(self, machine_id: int, serial: str) -> None:
-        result = self.execute(
+    async def set_machine_serial(self, machine_id: int, serial: str) -> None:
+        result = await self.execute(
             machine.update(machine.c.id == machine_id).values(keychip=serial)
         )
         if result is None:
@@ -126,8 +125,8 @@ class ArcadeData(BaseData):
             )
         return result.lastrowid
 
-    def set_machine_boardid(self, machine_id: int, boardid: str) -> None:
-        result = self.execute(
+    async def set_machine_boardid(self, machine_id: int, boardid: str) -> None:
+        result = await self.execute(
             machine.update(machine.c.id == machine_id).values(board=boardid)
         )
         if result is None:
@@ -135,21 +134,21 @@ class ArcadeData(BaseData):
                 f"Failed to update board id for machine {machine_id} -> {boardid}"
             )
 
-    def get_arcade(self, id: int) -> Optional[Row]:
+    async def get_arcade(self, id: int) -> Optional[Row]:
         sql = arcade.select(arcade.c.id == id)
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchone()
     
-    def get_arcade_machines(self, id: int) -> Optional[List[Row]]:
+    async def get_arcade_machines(self, id: int) -> Optional[List[Row]]:
         sql = machine.select(machine.c.arcade == id)
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchall()
 
-    def put_arcade(
+    async def put_arcade(
         self,
         name: str,
         nickname: str = None,
@@ -172,42 +171,42 @@ class ArcadeData(BaseData):
             regional_id=regional_id,
         )
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.lastrowid
 
-    def get_arcades_managed_by_user(self, user_id: int) -> Optional[List[Row]]:
+    async def get_arcades_managed_by_user(self, user_id: int) -> Optional[List[Row]]:
         sql = select(arcade).join(arcade_owner, arcade_owner.c.arcade == arcade.c.id).where(arcade_owner.c.user == user_id)
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return False
         return result.fetchall()
     
-    def get_manager_permissions(self, user_id: int, arcade_id: int) -> Optional[int]:
+    async def get_manager_permissions(self, user_id: int, arcade_id: int) -> Optional[int]:
         sql = select(arcade_owner.c.permissions).where(and_(arcade_owner.c.user == user_id, arcade_owner.c.arcade == arcade_id))
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return False
         return result.fetchone()
 
-    def get_arcade_owners(self, arcade_id: int) -> Optional[Row]:
+    async def get_arcade_owners(self, arcade_id: int) -> Optional[Row]:
         sql = select(arcade_owner).where(arcade_owner.c.arcade == arcade_id)
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchall()
 
-    def add_arcade_owner(self, arcade_id: int, user_id: int) -> None:
+    async def add_arcade_owner(self, arcade_id: int, user_id: int) -> None:
         sql = insert(arcade_owner).values(arcade=arcade_id, user=user_id)
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.lastrowid
 
-    def format_serial(
+    async def format_serial(
         self, platform_code: str, platform_rev: int, serial_num: int, append: int = 4152
     ) -> str:
         return f"{platform_code}{platform_rev:02d}A{serial_num:04d}{append:04d}"  # 0x41 = A, 0x52 = R
@@ -218,9 +217,16 @@ class ArcadeData(BaseData):
         
         return True
 
-    def find_arcade_by_name(self, name: str) -> List[Row]:
+    async def get_arcade_by_name(self, name: str) -> Optional[List[Row]]:
         sql = arcade.select(or_(arcade.c.name.like(f"%{name}%"), arcade.c.nickname.like(f"%{name}%")))
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
-            return False
+            return None
+        return result.fetchall()
+
+    async def get_arcades_by_ip(self, ip: str) -> Optional[List[Row]]:
+        sql = arcade.select().where(arcade.c.ip == ip)
+        result = await self.execute(sql)
+        if result is None:
+            return None
         return result.fetchall()

core/data/schema/base.py

@@ -15,14 +15,6 @@ from core.config import CoreConfig
 
 metadata = MetaData()
 
-schema_ver = Table(
-    "schema_versions",
-    metadata,
-    Column("game", String(4), primary_key=True, nullable=False),
-    Column("version", Integer, nullable=False, server_default="1"),
-    mysql_charset="utf8mb4",
-)
-
 event_log = Table(
     "event_log",
     metadata,
@@ -43,11 +35,11 @@ class BaseData:
         self.conn = conn
         self.logger = logging.getLogger("database")
 
-    def execute(self, sql: str, opts: Dict[str, Any] = {}) -> Optional[CursorResult]:
+    async def execute(self, sql: str, opts: Dict[str, Any] = {}) -> Optional[CursorResult]:
         res = None
 
         try:
-            self.logger.info(f"SQL Execute: {''.join(str(sql).splitlines())}")
+            self.logger.debug(f"SQL Execute: {''.join(str(sql).splitlines())}")
             res = self.conn.execute(text(sql), opts)
 
         except SQLAlchemyError as e:
@@ -82,52 +74,7 @@ class BaseData:
         """
         return randrange(10000, 9999999)
 
-    def get_all_schema_vers(self) -> Optional[List[Row]]:
-        sql = select(schema_ver)
-
-        result = self.execute(sql)
-        if result is None:
-            return None
-        return result.fetchall()
-
-    def get_schema_ver(self, game: str) -> Optional[int]:
-        sql = select(schema_ver).where(schema_ver.c.game == game)
-
-        result = self.execute(sql)
-        if result is None:
-            return None
-
-        row = result.fetchone()
-        if row is None:
-            return None
-
-        return row["version"]
-    
-    def touch_schema_ver(self, ver: int, game: str = "CORE") -> Optional[int]:
-        sql = insert(schema_ver).values(game=game, version=ver)
-        conflict = sql.on_duplicate_key_update(version=schema_ver.c.version)
-
-        result = self.execute(conflict)
-        if result is None:
-            self.logger.error(
-                f"Failed to update schema version for game {game} (v{ver})"
-            )
-            return None
-        return result.lastrowid
-
-    def set_schema_ver(self, ver: int, game: str = "CORE") -> Optional[int]:
-        sql = insert(schema_ver).values(game=game, version=ver)
-        conflict = sql.on_duplicate_key_update(version=ver)
-
-        result = self.execute(conflict)
-        if result is None:
-            self.logger.error(
-                f"Failed to update schema version for game {game} (v{ver})"
-            )
-            return None
-        return result.lastrowid
-
-    def log_event(
+    async def log_event(
         self, system: str, type: str, severity: int, message: str, details: Dict = {}
     ) -> Optional[int]:
         sql = event_log.insert().values(
@@ -137,7 +84,7 @@ class BaseData:
             message=message,
             details=json.dumps(details),
         )
-        result = self.execute(sql)
+        result = await self.execute(sql)
 
         if result is None:
             self.logger.error(
@@ -147,9 +94,9 @@ class BaseData:
 
         return result.lastrowid
 
-    def get_event_log(self, entries: int = 100) -> Optional[List[Dict]]:
+    async def get_event_log(self, entries: int = 100) -> Optional[List[Dict]]:
         sql = event_log.select().limit(entries).all()
-        result = self.execute(sql)
+        result = await self.execute(sql)
 
         if result is None:
             return None
@@ -157,6 +104,8 @@ class BaseData:
 
     def fix_bools(self, data: Dict) -> Dict:
         for k, v in data.items():
+            if k == "userName" or k == "teamName":
+                continue
             if type(v) == str and v.lower() == "true":
                 data[k] = True
             elif type(v) == str and v.lower() == "false":

core/data/schema/card.py

@@ -27,70 +27,101 @@ aime_card = Table(
 
 
 class CardData(BaseData):
-    def get_card_by_access_code(self, access_code: str) -> Optional[Row]:
+    async def get_card_by_access_code(self, access_code: str) -> Optional[Row]:
         sql = aime_card.select(aime_card.c.access_code == access_code)
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchone()
 
-    def get_card_by_id(self, card_id: int) -> Optional[Row]:
+    async def get_card_by_id(self, card_id: int) -> Optional[Row]:
         sql = aime_card.select(aime_card.c.id == card_id)
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchone()
 
-    def update_access_code(self, old_ac: str, new_ac: str) -> None:
+    async def update_access_code(self, old_ac: str, new_ac: str) -> None:
         sql = aime_card.update(aime_card.c.access_code == old_ac).values(
             access_code=new_ac
         )
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             self.logger.error(
                 f"Failed to change card access code from {old_ac} to {new_ac}"
             )
 
-    def get_user_id_from_card(self, access_code: str) -> Optional[int]:
+    async def get_user_id_from_card(self, access_code: str) -> Optional[int]:
         """
         Given a 20 digit access code as a string, get the user id associated with that card
         """
-        card = self.get_card_by_access_code(access_code)
+        card = await self.get_card_by_access_code(access_code)
         if card is None:
             return None
 
         return int(card["user"])
 
-    def delete_card(self, card_id: int) -> None:
+    async def get_card_banned(self, access_code: str) -> Optional[bool]:
+        """
+        Given a 20 digit access code as a string, check if the card is banned
+        """
+        card = await self.get_card_by_access_code(access_code)
+        if card is None:
+            return None
+        if card["is_banned"]:
+            return True
+        return False
+    
+    async def get_card_locked(self, access_code: str) -> Optional[bool]:
+        """
+        Given a 20 digit access code as a string, check if the card is locked
+        """
+        card = await self.get_card_by_access_code(access_code)
+        if card is None:
+            return None
+        if card["is_locked"]:
+            return True
+        return False
+
+    async def delete_card(self, card_id: int) -> None:
         sql = aime_card.delete(aime_card.c.id == card_id)
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             self.logger.error(f"Failed to delete card with id {card_id}")
 
-    def get_user_cards(self, aime_id: int) -> Optional[List[Row]]:
+    async def get_user_cards(self, aime_id: int) -> Optional[List[Row]]:
         """
         Returns all cards owned by a user
         """
         sql = aime_card.select(aime_card.c.user == aime_id)
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchall()
 
-    def create_card(self, user_id: int, access_code: str) -> Optional[int]:
+    async def create_card(self, user_id: int, access_code: str) -> Optional[int]:
         """
         Given a aime_user id and a 20 digit access code as a string, create a card and return the ID if successful
         """
         sql = aime_card.insert().values(user=user_id, access_code=access_code)
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.lastrowid
 
+    async def update_card_last_login(self, access_code: str) -> None:
+        sql = aime_card.update(aime_card.c.access_code == access_code).values(
+            last_login_date=func.now()
+        )
+        
+        result = await self.execute(sql)
+        if result is None:
+            self.logger.warn(f"Failed to update last login time for {access_code}")
+    
     def to_access_code(self, luid: str) -> str:
         """
         Given a felica cards internal 16 hex character luid, convert it to a 0-padded 20 digit access code as a string

core/data/schema/user.py

@@ -1,4 +1,3 @@
-from enum import Enum
 from typing import Optional, List
 from sqlalchemy import Table, Column
 from sqlalchemy.types import Integer, String, TIMESTAMP
@@ -24,15 +23,8 @@ aime_user = Table(
     mysql_charset="utf8mb4",
 )
 
-
-class PermissionBits(Enum):
-    PermUser = 1
-    PermMod = 2
-    PermSysAdmin = 4
-
-
 class UserData(BaseData):
-    def create_user(
+    async def create_user(
         self,
         id: int = None,
         username: str = None,
@@ -60,20 +52,20 @@ class UserData(BaseData):
             username=username, email=email, password=password, permissions=permission
         )
 
-        result = self.execute(conflict)
+        result = await self.execute(conflict)
         if result is None:
             return None
         return result.lastrowid
 
-    def get_user(self, user_id: int) -> Optional[Row]:
+    async def get_user(self, user_id: int) -> Optional[Row]:
         sql = select(aime_user).where(aime_user.c.id == user_id)
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return False
         return result.fetchone()
 
-    def check_password(self, user_id: int, passwd: bytes = None) -> bool:
-        usr = self.get_user(user_id)
+    async def check_password(self, user_id: int, passwd: bytes = None) -> bool:
+        usr = await self.get_user(user_id)
         if usr is None:
             return False
 
@@ -85,39 +77,46 @@ class UserData(BaseData):
 
         return bcrypt.checkpw(passwd, usr["password"].encode())
 
-    def reset_autoincrement(self, ai_value: int) -> None:
-        # ALTER TABLE isn't in sqlalchemy so we do this the ugly way
-        sql = f"ALTER TABLE aime_user AUTO_INCREMENT={ai_value}"
-        self.execute(sql)
-
-    def delete_user(self, user_id: int) -> None:
+    async def delete_user(self, user_id: int) -> None:
         sql = aime_user.delete(aime_user.c.id == user_id)
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             self.logger.error(f"Failed to delete user with id {user_id}")
 
-    def get_unregistered_users(self) -> List[Row]:
+    async def get_unregistered_users(self) -> List[Row]:
         """
         Returns a list of users who have not registered with the webui. They may or may not have cards.
         """
         sql = select(aime_user).where(aime_user.c.password == None)
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchall()
 
-    def find_user_by_email(self, email: str) -> Row:
+    async def find_user_by_email(self, email: str) -> Row:
         sql = select(aime_user).where(aime_user.c.email == email)
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return False
         return result.fetchone()
 
-    def find_user_by_username(self, username: str) -> List[Row]:
+    async def find_user_by_username(self, username: str) -> List[Row]:
         sql = aime_user.select(aime_user.c.username.like(f"%{username}%"))
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return False
         return result.fetchall()
+
+    async def change_password(self, user_id: int, new_passwd: str) -> bool:
+        sql = aime_user.update(aime_user.c.id == user_id).values(password = new_passwd)
+
+        result = await self.execute(sql)
+        return result is not None
+
+    async def change_username(self, user_id: int, new_name: str) -> bool:
+        sql = aime_user.update(aime_user.c.id == user_id).values(username = new_name)
+
+        result = await self.execute(sql)
+        return result is not None

core/data/schema/versions/SBZV_5_rollback.sql

@@ -0,0 +1,2 @@
+ALTER TABLE diva_profile
+    DROP skn_eqp;

core/data/schema/versions/SBZV_6_upgrade.sql

@@ -0,0 +1,2 @@
+ALTER TABLE diva_profile
+    ADD skn_eqp INT NOT NULL DEFAULT 0;

core/data/schema/versions/SDBT_4_rollback.sql

@@ -0,0 +1,12 @@
+SET FOREIGN_KEY_CHECKS = 0;
+
+ALTER TABLE chuni_score_playlog
+  CHANGE COLUMN isClear isClear TINYINT(1) NULL DEFAULT NULL;
+
+ALTER TABLE chuni_score_best
+  CHANGE COLUMN isSuccess isSuccess TINYINT(1) NULL DEFAULT NULL ;
+
+ALTER TABLE chuni_score_playlog
+  DROP COLUMN ticketId;
+
+SET FOREIGN_KEY_CHECKS = 1;

core/data/schema/versions/SDBT_5_upgrade.sql

@@ -0,0 +1,12 @@
+SET FOREIGN_KEY_CHECKS = 0;
+
+ALTER TABLE chuni_score_playlog
+  CHANGE COLUMN isClear isClear TINYINT(6) NULL DEFAULT NULL;
+
+ALTER TABLE chuni_score_best
+  CHANGE COLUMN isSuccess isSuccess INT(11) NULL DEFAULT NULL ;
+
+ALTER TABLE chuni_score_playlog
+  ADD COLUMN ticketId INT(11) NULL AFTER machineType;
+
+SET FOREIGN_KEY_CHECKS = 1;

core/data/schema/versions/SDDT_5_rollback.sql

@@ -0,0 +1,22 @@
+SET FOREIGN_KEY_CHECKS=0;
+
+ALTER TABLE ongeki_user_event_point DROP COLUMN version;
+ALTER TABLE ongeki_user_event_point DROP COLUMN `rank`;
+ALTER TABLE ongeki_user_event_point DROP COLUMN `type`;
+ALTER TABLE ongeki_user_event_point DROP COLUMN date;
+
+ALTER TABLE ongeki_user_tech_event DROP COLUMN version;
+
+ALTER TABLE ongeki_user_mission_point DROP COLUMN version;
+
+ALTER TABLE ongeki_static_events DROP COLUMN endDate;
+
+DROP TABLE ongeki_tech_event_ranking;
+DROP TABLE ongeki_static_music_ranking_list;
+DROP TABLE ongeki_static_rewards;
+DROP TABLE ongeki_static_present_list;
+DROP TABLE ongeki_static_tech_music;
+DROP TABLE ongeki_static_client_testmode;
+DROP TABLE ongeki_static_game_point;
+
+SET FOREIGN_KEY_CHECKS=1;

core/data/schema/versions/SDDT_6_upgrade.sql

@@ -0,0 +1,98 @@
+SET FOREIGN_KEY_CHECKS=0;
+
+ALTER TABLE ongeki_user_event_point ADD COLUMN version INTEGER NOT NULL;
+ALTER TABLE ongeki_user_event_point ADD COLUMN `rank` INTEGER;
+ALTER TABLE ongeki_user_event_point ADD COLUMN `type` INTEGER NOT NULL;
+ALTER TABLE ongeki_user_event_point ADD COLUMN date VARCHAR(25);
+
+ALTER TABLE ongeki_user_tech_event ADD COLUMN version INTEGER NOT NULL;
+
+ALTER TABLE ongeki_user_mission_point ADD COLUMN version INTEGER NOT NULL;
+
+ALTER TABLE ongeki_static_events ADD COLUMN endDate TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP;
+
+CREATE TABLE ongeki_tech_event_ranking (
+    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
+    user INT NOT NULL,
+    version INT NOT NULL,
+    date VARCHAR(25),
+    eventId INT NOT NULL,
+    `rank` INT,
+    totalPlatinumScore INT NOT NULL,
+    totalTechScore INT NOT NULL,
+    UNIQUE KEY ongeki_tech_event_ranking_uk (user, eventId),
+    CONSTRAINT ongeki_tech_event_ranking_ibfk1 FOREIGN KEY (user) REFERENCES aime_user(id) ON DELETE CASCADE ON UPDATE CASCADE
+);
+
+CREATE TABLE ongeki_static_music_ranking_list (
+    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
+    version INT NOT NULL,
+    musicId INT NOT NULL,
+    point INT NOT NULL,
+    userName VARCHAR(255),
+    UNIQUE KEY ongeki_static_music_ranking_list_uk (version, musicId)
+);
+
+CREATE TABLE ongeki_static_rewards (
+    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
+    version INT NOT NULL,
+    rewardId INT NOT NULL,
+    rewardName VARCHAR(255) NOT NULL,
+    itemKind INT NOT NULL,
+    itemId INT NOT NULL,
+    UNIQUE KEY ongeki_tech_event_ranking_uk (version, rewardId)
+);
+
+CREATE TABLE ongeki_static_present_list (
+    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
+    version INT NOT NULL,
+    presentId INT NOT NULL,
+    presentName VARCHAR(255) NOT NULL,
+    rewardId INT NOT NULL,
+    stock INT NOT NULL,
+    message VARCHAR(255),
+    startDate VARCHAR(25) NOT NULL,
+    endDate VARCHAR(25) NOT NULL,
+    UNIQUE KEY ongeki_static_present_list_uk (version, presentId, rewardId)
+);
+
+CREATE TABLE ongeki_static_tech_music (
+    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
+    version INT NOT NULL,
+    eventId INT NOT NULL,
+    musicId INT NOT NULL,
+    level INT NOT NULL,
+    UNIQUE KEY ongeki_static_tech_music_uk (version, musicId, eventId)
+);
+
+CREATE TABLE ongeki_static_client_testmode (
+    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
+    regionId INT NOT NULL,
+    placeId INT NOT NULL,
+    clientId VARCHAR(11) NOT NULL,
+    updateDate TIMESTAMP NOT NULL,
+    isDelivery BOOLEAN NOT NULL,
+    groupId INT NOT NULL,
+    groupRole INT NOT NULL,
+    continueMode INT NOT NULL,
+    selectMusicTime INT NOT NULL,
+    advertiseVolume INT NOT NULL,
+    eventMode INT NOT NULL,
+    eventMusicNum INT NOT NULL,
+    patternGp INT NOT NULL,
+    limitGp INT NOT NULL,
+    maxLeverMovable INT NOT NULL,
+    minLeverMovable INT NOT NULL,
+    UNIQUE KEY ongeki_static_client_testmode_uk (clientId)
+);
+
+CREATE TABLE ongeki_static_game_point (
+    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
+    `type` INT NOT NULL,
+    cost INT NOT NULL,
+    startDate VARCHAR(25) NOT NULL DEFAULT "2000-01-01 05:00:00.0",
+    endDate VARCHAR(25) NOT NULL DEFAULT "2099-01-01 05:00:00.0",
+    UNIQUE KEY ongeki_static_game_point_uk (`type`)
+);
+
+SET FOREIGN_KEY_CHECKS=1;

core/data/schema/versions/SDEZ_7_rollback.sql

@@ -0,0 +1,10 @@
+ALTER TABLE mai2_profile_detail
+    DROP COLUMN mapStock;
+
+ALTER TABLE mai2_profile_extend
+    DROP COLUMN selectResultScoreViewType;
+
+ALTER TABLE mai2_profile_option
+    DROP COLUMN outFrameType,
+    DROP COLUMN touchVolume,
+    DROP COLUMN breakSlideVolume;

core/data/schema/versions/SDEZ_8_upgrade.sql

@@ -0,0 +1,10 @@
+ALTER TABLE mai2_profile_detail
+    ADD mapStock INT NULL AFTER playCount;
+
+ALTER TABLE mai2_profile_extend
+    ADD selectResultScoreViewType INT NULL AFTER selectResultDetails;
+
+ALTER TABLE mai2_profile_option
+    ADD outFrameType INT NULL AFTER dispCenter,
+    ADD touchVolume INT NULL AFTER slideVolume,
+    ADD breakSlideVolume INT NULL AFTER slideVolume;

core/frontend/arcade/index.jinja

@@ -1,4 +0,0 @@
-{% extends "core/frontend/index.jinja" %}
-{% block content %}
-<h1>{{ arcade.name }}</h1>
-{% endblock content %}

core/frontend/gate/create.jinja

@@ -1,44 +0,0 @@
-{% extends "core/frontend/index.jinja" %}
-{% block content %}
-<div class="row justify-content-md-center form-signin">
-    <div class="col col-lg-4">
-        <form id="create" action="/gate/gate.create" method="post">
-            <h1 class="h3 mb-3 fw-normal">Sign-up</h1>
-
-            <div class="form-floating mb-3">
-                <input class="form-control" name="access-code" id="access-code" type="text"
-                    placeholder="00000000000000000000" value={{ code }} maxlength="20" required>
-                <label for="access-code">Access Code*</label>
-            </div>
-            <div class="form-floating mb-3">
-                <input type="text" class="form-control" name="username" id="username" placeholder="username">
-                <label for="username">Username*</label>
-            </div>
-            <div class="form-floating mb-3">
-                <input type="email" class="form-control" name="email" id="email" placeholder="example@example.com">
-                <label for="email">E-Mail*</label>
-            </div>
-            <div class="form-floating mb-3">
-                <input type="password" class="form-control" name="passwd" id="passwd" placeholder="Password">
-                <label for="passwd">Password*</label>
-            </div>
-            
-            <!--
-            <div class="form-check text-start my-3">
-                <input class="form-check-input" type="checkbox" value="remember-me" id="remember-me">
-                <label class="form-check-label" for="remember-me">
-                    Remember me
-                </label>
-            </div>
-            -->
-
-            <div class="alert alert-warning" role="alert">
-                If you have not registered a card with this server, you cannot create a WebUI account!
-            </div>
-
-            <button class="btn btn-primary w-100 py-2" type="submit">Sign-up</button>
-            <!-- <p class="mt-5 mb-3 text-body-secondary">© 2023 ARTEMiS</p>-->
-        </form>
-    </div>
-</div>
-{% endblock content %}

core/frontend/gate/gate.jinja

@@ -1,52 +0,0 @@
-{% extends "core/frontend/index.jinja" %}
-{% block content %}
-<style>
-    .form-signin input[type="text"] {
-        margin-bottom: -1px;
-        border-bottom-right-radius: 0;
-        border-bottom-left-radius: 0;
-    }
-
-    .form-signin input[type="password"] {
-        margin-bottom: 10px;
-        border-top-left-radius: 0;
-        border-top-right-radius: 0;
-    }
-</style>
-<div class="row justify-content-md-center form-signin">
-    <div class="col col-lg-4">
-        <form id="login" action="/gate/gate.login" method="post">
-            <h1 class="h3 mb-3 fw-normal">Login</h1>
-
-            {% include "core/frontend/widgets/err_banner.jinja" %}
-
-            <div class="form-floating">
-                <input form="login" class="form-control" name="access-code" id="access-code" type="text"
-                    placeholder="00000000000000000000" maxlength="20" required>
-                <label for="access-code">Access Code</label>
-            </div>
-            <div class="form-floating">
-                <input form="login" type="password" class="form-control" name="passwd" id="passwd" placeholder="Password">
-                <label for="passwd">Password</label>
-            </div>
-            
-            <!--
-            <div class="form-check text-start my-3">
-                <input class="form-check-input" type="checkbox" value="remember-me" id="remember-me">
-                <label class="form-check-label" for="remember-me">
-                    Remember me
-                </label>
-            </div>
-            -->
-
-            <div class="alert alert-info" role="alert">
-                To register for the WebUI, type in the <strong>access code</strong> of your card, as shown in a game, and leave the
-                password field blank.
-            </div>
-
-            <button class="btn btn-primary w-100 py-2" type="submit">Sign in</button>
-            <!-- <p class="mt-5 mb-3 text-body-secondary">© 2023 ARTEMiS</p>-->
-        </form>
-    </div>
-</div>
-{% endblock content %}

core/frontend/index.jinja

@@ -1,25 +0,0 @@
-<!DOCTYPE html>
-<html lang="en" data-bs-theme="dark">
-
-<head>
-    <meta charset="utf-8">
-    <meta name="viewport" content="width=device-width, initial-scale=1">
-    <title>{{ title }}</title>
-    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.1/dist/css/bootstrap.min.css" rel="stylesheet"
-        integrity="sha384-4bw+/aepP/YC94hEpVNVgiZdgIC5+VKNBQNGCHeKRQN+PtmoHDEXuppvnDJzQIu9" crossorigin="anonymous">
-    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.10.5/font/bootstrap-icons.css">
-</head>
-
-<body>
-    {% include "core/frontend/widgets/topbar.jinja" %}
-    <div class="container">
-        {% block content %}
-        <h1>{{ server_name }}</h1>
-        {% endblock content %}
-    </div>
-    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.1/dist/js/bootstrap.bundle.min.js"
-        integrity="sha384-HwwvtgBNo3bZJJLYd8oVXjrBZt8cqVSpeBNS5n7C8IVInixGAoxmnlMuBnhbgrkm"
-        crossorigin="anonymous"></script>
-</body>
-
-</html>

core/frontend/machine/index.jinja

@@ -1,5 +0,0 @@
-{% extends "core/frontend/index.jinja" %}
-{% block content %}
-{% include "core/frontend/widgets/err_banner.jinja" %}
-<h1>Machine Management</h1>
-{% endblock content %}

core/frontend/sys/index.jinja

@@ -1,98 +0,0 @@
-{% extends "core/frontend/index.jinja" %}
-{% block content %}
-<h1>System Management</h1>
-
-<div class="row" id="rowForm">
-    {% if sesh.permissions >= 2 %}
-    <div class="col-sm-6" style="max-width: 25%;">
-        <form id="usrLookup" name="usrLookup" action="/sys/lookup.user" class="form-inline">
-            <h3>User Search</h3>
-            <div class="form-group">
-                <label for="usrEmail">Email address</label>
-                <input type="email" class="form-control" id="usrEmail" name="usrEmail" aria-describedby="emailHelp">
-            </div>
-            OR
-            <div class="form-group">
-                <label for="usrName">Username</label>
-                <input type="text" class="form-control" id="usrName" name="usrName">
-            </div>
-            OR
-            <div class="form-group">
-                <label for="usrId">User ID</label>
-                <input type="number" class="form-control" id="usrId" name="usrId">
-            </div>
-            <br />
-            <button type="submit" class="btn btn-primary">Search</button>
-        </form>
-    </div>
-    {% endif %}
-    {% if sesh.permissions >= 4 %}
-    <div class="col-sm-6" style="max-width: 25%;">
-        <form id="arcadeLookup" name="arcadeLookup" action="/sys/lookup.arcade" class="form-inline" >
-            <h3>Arcade Search</h3>
-            <div class="form-group">
-                <label for="arcadeName">Arcade Name</label>
-                <input type="text" class="form-control" id="arcadeName" name="arcadeName">
-            </div>
-            OR
-            <div class="form-group">
-                <label for="arcadeId">Arcade ID</label>
-                <input type="number" class="form-control" id="arcadeId" name="arcadeId">
-            </div>
-            OR
-            <div class="form-group">
-                <label for="arcadeUser">Owner User ID</label>
-                <input type="number" class="form-control" id="arcadeUser" name="arcadeUser">
-            </div>
-            <br />
-            <button type="submit" class="btn btn-primary">Search</button>
-        </form>
-    </div>
-    <div class="col-sm-6" style="max-width: 25%;">
-        <form id="cabLookup" name="cabLookup" action="/sys/lookup.cab" class="form-inline" >
-            <h3>Machine Search</h3>
-            <div class="form-group">
-                <label for="cabSerial">Machine Serial</label>
-                <input type="text" class="form-control" id="cabSerial" name="cabSerial">
-            </div>
-            OR
-            <div class="form-group">
-                <label for="cabId">Machine ID</label>
-                <input type="number" class="form-control" id="cabId" name="cabId">
-            </div>
-            OR
-            <div class="form-group">
-                <label for="cabAcId">Arcade ID</label>
-                <input type="number" class="form-control" id="cabAcId" name="cabAcId">
-            </div>
-            <br />
-            <button type="submit" class="btn btn-primary">Search</button>
-        </form>
-    </div>
-    {% endif %}
-</div>
-<div class="row" id="rowResult" style="margin: 10px;">
-    {% if sesh.permissions >= 2 %}
-    <div id="userSearchResult" class="col-sm-6" style="max-width: 25%;">
-        {% for usr in usrlist %}
-        <pre><a href=/user/{{ usr.id }}>{{ usr.id }} | {{ usr.username }}</a></pre>
-        {% endfor %}
-    </div>
-    {% endif %}
-    {% if sesh.permissions >= 4 %}
-    <div id="arcadeSearchResult" class="col-sm-6" style="max-width: 25%;">
-        {% for ac in aclist %}
-        <pre><a href=/arcade/{{ ac.id }}>{{ ac.id }} | {{ ac.name }}</a></pre>
-        {% endfor %}
-    </div
-    ><div id="cabSearchResult" class="col-sm-6" style="max-width: 25%;">
-        {% for cab in cablist %}
-        <a href=/cab/{{ cab.id }}><pre>{{ cab.id }} | {{ cab.game if cab.game is defined else "ANY "  }} | {{ cab.serial }}</pre></a>
-        {% endfor %}
-    </div>
-    {% endif %}
-</div>
-<div class="row" id="rowAdd">
-
-</div>
-{% endblock content %}

core/frontend/user/index.jinja

@@ -1,126 +0,0 @@
-{% extends "core/frontend/index.jinja" %}
-{% block content %}
-<h1>{{ sesh["username"] }}'s Account</h1>
-
-<div class="card mb-3">
-    <div class="card-body">
-        <h3 class="card-title">Cards</h3>
-        <!--<h4 class="card-subtitle mb-2 text-body-secondary">Card subtitle</h4>-->
-        <p class="card-text">All aime cards are listed here for the given user.</p>
-        <a href="#" data-bs-toggle="modal" data-bs-target="#card-add" class="btn btn-primary mb-3">Add Card</a>
-        {% if cards is defined and cards|length > 0 %}
-        <div class="container">
-            <div class="row">
-                <div class="col-12">
-                    <table class="table table-hover">
-                        <thead>
-                            <tr>
-                                <th scope="col">#</th>
-                                <th scope="col">Access Code</th>
-                                <th scope="col">Status</th>
-                                <th scope="col">Actions</th>
-                            </tr>
-                        </thead>
-                        <tbody>
-                            {% for c in cards %}
-                            <tr class="align-middle">
-                                <th scope="row">{{ c.index }}</th>
-                                <td>{{ c.access_code }}</td>
-                                <td>
-                                    {% if c.status == 'Active'%}
-                                    <span class="badge rounded-pill text-bg-success">Active</span>
-                                    {% elif c.status == 'Locked' %}
-                                    <span class="badge rounded-pill text-bg-warning">Locked</span>
-                                    {% endif %}
-                                </td>
-                                <td>
-                                    {% if c.status == 'Active'%}
-                                    <button type="button" class="btn btn-warning btn-sm">Lock</i></button>
-                                    {% elif c.status == 'Locked' %}
-                                    <button type="button" class="btn btn-success btn-sm">Unlock</i></button>
-                                    {% endif %}
-                                    <button type="button" class="btn btn-danger btn-sm">Delete</i></button>
-                                </td>
-                            </tr>
-                            {% endfor %}
-                        </tbody>
-                    </table>
-                </div>
-            </div>
-        </div>
-        {% else %}
-        <div class="alert alert-warning" role="alert">
-            You have no cards registered to your account. Please add one.
-        </div>
-        {% endif %}
-        <!--<a href="#" data-bs-toggle="modal" data-bs-target="#card-add" class="card-link">Add Card</a>-->
-    </div>
-</div>
-
-<div class="card">
-    <div class="card-body">
-        <h3 class="card-title">Arcades</h3>
-        <!--<h4 class="card-subtitle mb-2 text-body-secondary">Card subtitle</h4>-->
-        <p class="card-text">All arcades connected to the given account are listed here.</p>
-        {% if arcades is defined and arcades|length > 0 %}
-        <div class="container">
-            <div class="row">
-                <div class="col-12">
-                    <table class="table table-hover">
-                        <thead>
-                            <tr>
-                                <th scope="col">#</th>
-                                <th scope="col">Name</th>
-                            </tr>
-                        </thead>
-                        <tbody>
-                            {% for a in arcades %}
-                            <tr class="align-middle clickable-row" data-href=/arcade/{{ a.id }}>
-                                <th scope="row">{{ a.index }}</th>
-                                <td>{{ a.name }}</td>
-                            </tr>
-                            {% endfor %}
-                        </tbody>
-                    </table>
-                </div>
-            </div>
-        </div>
-        {% else %}
-        <div class="alert alert-info" role="alert">
-            You have no arcades connected to your account.
-        </div>
-        {% endif %}
-    </div>
-</div>
-
-<div class="modal fade" id="card-add" tabindex="-1" aria-labelledby="card-add-label" aria-hidden="true">
-    <div class="modal-dialog">
-        <div class="modal-content">
-            <div class="modal-header">
-                <h1 class="modal-title fs-5" id="card-add-label">Add Card</h1>
-                <button type="button" class="btn-close" data-bs-dismiss="modal" aria-label="Close"></button>
-            </div>
-            <div class="modal-body">
-                <h3 class="fs-5">How to:</h3>
-                Scan your card on any networked game and press the "View Access Code" button (varies by game) and enter
-                the 20
-                digit code below.
-                <hr>
-                FOR AMUSE IC CARDS: DO NOT ENTER THE CODE SHOWN ON THE BACK OF THE CARD ITSELF OR IT WILL NOT WORK!
-                <form>
-                    <div class="form-floating mt-3">
-                        <label for="access-code" class="col-form-label">Access Code</label>
-                        <input maxlength="20" type="text" required pattern="[0-9]{20}" class="form-control"
-                            id="access-code">
-                    </div>
-                </form>
-            </div>
-            <div class="modal-footer">
-                <button type="button" class="btn btn-secondary" data-bs-dismiss="modal">Close</button>
-                <button type="button" class="btn btn-primary">Add</button>
-            </div>
-        </div>
-    </div>
-</div>
-
-{% endblock content %}

core/frontend/widgets/err_banner.jinja

@@ -1,17 +0,0 @@
-{% if error > 0 %}
-<div class="alert alert-danger" role="alert">
-{% if error == 1 %}
-Card not registered, or wrong password
-{% elif error == 2 %}
-Missing or malformed access code
-{% elif error == 3 %}
-Failed to create user
-{% elif error == 4 %}
-Arcade not found
-{% elif error == 5 %}
-Machine not found
-{% else %}
-An unknown error occoured
-{% endif %}
-</div>
-{% endif %}

core/frontend/widgets/topbar.jinja

@@ -1,56 +0,0 @@
-<nav class="p-3 mb-3 navbar navbar-expand-lg bg-body-tertiary" aria-label="Thirteenth navbar example">
-    <div class="container">
-        <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarsExample11"
-            aria-controls="navbarsExample11" aria-expanded="false" aria-label="Toggle navigation">
-            <span class="navbar-toggler-icon"></span>
-        </button>
-
-        <div class="collapse navbar-collapse d-lg-flex" id="navbarsExample11">
-            <a class="navbar-brand col-lg-3 me-0" href= />ARTEMiS</a>
-            <ul class="navbar-nav col-lg-6 justify-content-lg-center nav-pills">
-                <li class="nav-item">
-                    <a class="nav-link {% if active_page == '/' %}active{% endif %}" aria-current="page" href= />Home</a>
-                </li>
-                {% for game in game_list %}
-                <li class="nav-item">
-                    <a class="nav-link {% if active_page == game.url %}active{% endif %}" href=/game/{{ game.url }}>{{ game.name }}</a>
-                </li>
-                {% endfor %}
-                <!--
-                    <li class="nav-item dropdown">
-                        <a class="nav-link dropdown-toggle" href="#" data-bs-toggle="dropdown"
-                            aria-expanded="false">Dropdown</a>
-                        <ul class="dropdown-menu">
-                            <li><a class="dropdown-item" href="#">Action</a></li>
-                            <li><a class="dropdown-item" href="#">Another action</a></li>
-                            <li><a class="dropdown-item" href="#">Something else here</a></li>
-                        </ul>
-                    </li>
-                    -->
-            </ul>
-            <div class="d-lg-flex col-lg-3 justify-content-lg-end">
-                {% if sesh is defined and sesh["user_id"] > 0 %}
-                <div class="btn-group dropdown">
-                    <a href="#" class="d-block link-body-emphasis text-decoration-none dropdown-toggle"
-                        data-bs-toggle="dropdown" aria-expanded="false">
-                       {{ sesh["username"] }}
-                    </a>
-                    <ul class="dropdown-menu text-small">
-                        {% if sesh["permissions"] >= 2 %}
-                        <li><a class="dropdown-item {% if active_page == 'sys' %}active{% endif %}" href="/sys">System</a></li>
-                        {% endif %}
-                        <li><a class="dropdown-item" href="#">Settings</a></li>
-                        <li><a class="dropdown-item {% if active_page == 'user' %}active{% endif %}" href="/user">Account</a></li>
-                        <li>
-                            <hr class="dropdown-divider">
-                        </li>
-                        <li><a class="dropdown-item" href="#">Sign out</a></li>
-                    </ul>
-                </div>
-                {% else %}
-                <a href="/gate"><button class="btn {% if active_page == 'gate' %}btn-primary{% else %}btn-outline-primary{% endif %}">Login</button></a>
-                {% endif %}
-            </div>
-        </div>
-    </div>
-</nav>

core/mucha.py

@@ -1,21 +1,23 @@
-from typing import Dict, Any, Optional, List
+from typing import Dict, Any, Optional
 import logging, coloredlogs
 from logging.handlers import TimedRotatingFileHandler
-from twisted.web import resource
-from twisted.web.http import Request
+from starlette.requests import Request
+from starlette.responses import PlainTextResponse
 from datetime import datetime
 from Crypto.Cipher import Blowfish
 import pytz
 
-from core import CoreConfig
-from core.utils import Utils
-
+from .config import CoreConfig
+from .utils import Utils
+from .title import TitleServlet
+from .data import Data
+from .const import *
 
 class MuchaServlet:
+    mucha_registry: Dict[str, Dict[str, str]] = {}
     def __init__(self, cfg: CoreConfig, cfg_dir: str) -> None:
         self.config = cfg
         self.config_dir = cfg_dir
-        self.mucha_registry: List[str] = []
 
         self.logger = logging.getLogger("mucha")
         log_fmt_str = "[%(asctime)s] Mucha | %(levelname)s | %(message)s"
@@ -37,91 +39,147 @@ class MuchaServlet:
         self.logger.setLevel(cfg.mucha.loglevel)
         coloredlogs.install(level=cfg.mucha.loglevel, logger=self.logger, fmt=log_fmt_str)
         
-        all_titles = Utils.get_all_titles()
+        self.data = Data(cfg)
 
-        for _, mod in all_titles.items():
-            if hasattr(mod, "index") and hasattr(mod.index, "get_mucha_info"):
-                enabled, game_cd = mod.index.get_mucha_info(
-                    self.config, self.config_dir
-                )
+        for _, mod in TitleServlet.title_registry.items():
+            enabled, game_cds, netids = mod.get_mucha_info(self.config, self.config_dir)
             if enabled:
-                    self.mucha_registry.append(game_cd)
+                for x in range(len(game_cds)):
+                    self.mucha_registry[game_cds[x]] = { "netid_prefix": netids[x] }
 
         self.logger.info(f"Serving {len(self.mucha_registry)} games")
 
-    def handle_boardauth(self, request: Request, _: Dict) -> bytes:
-        req_dict = self.mucha_preprocess(request.content.getvalue())
+    async def handle_boardauth(self, request: Request) -> bytes:
+        bod = await request.body()
+        req_dict = self.mucha_preprocess(bod)
         client_ip = Utils.get_ip_addr(request)
 
         if req_dict is None:
             self.logger.error(
-                f"Error processing mucha request {request.content.getvalue()}"
+                f"Error processing mucha request {bod}"
             )
-            return b"RESULTS=000"
+            return PlainTextResponse("RESULTS=000")
 
         req = MuchaAuthRequest(req_dict)
-        self.logger.info(f"Boardauth request from {client_ip} for {req.gameVer}")        
         self.logger.debug(f"Mucha request {vars(req)}")
         
-        if req.gameCd not in self.mucha_registry:
-            self.logger.warning(f"Unknown gameCd {req.gameCd}")
-            return b"RESULTS=000"
+        if not req.gameCd or not req.gameVer or not req.sendDate or not req.countryCd or not req.serialNum:
+            self.logger.warn(f"Missing required fields - {vars(req)}")
+            return PlainTextResponse("RESULTS=000")
+
+        minfo = self.mucha_registry.get(req.gameCd, {})
+
+        if not minfo:
+            self.logger.warning(f"Unknown gameCd {req.gameCd} from {client_ip}")
+            return PlainTextResponse("RESULTS=000")
 
-        # TODO: Decrypt S/N
         b_key = b""
         for x in range(8):
             b_key += req.sendDate[(x - 1) & 7].encode()
         
-        cipher = Blowfish.new(b_key, Blowfish.MODE_ECB)
-        sn_decrypt = cipher.decrypt(bytes.fromhex(req.serialNum))
-        self.logger.debug(f"Decrypt SN to {sn_decrypt.hex()}")
+        b_iv = b_key # what the fuck namco
+
+        cipher = Blowfish.new(b_key, Blowfish.MODE_CBC, b_iv)
+        try:
+            sn_decrypt = cipher.decrypt(bytes.fromhex(req.serialNum))[:12].decode()
+        except Exception as e:
+            self.logger.error(f"Decrypt SN {req.serialNum} failed! - {e}")
+            return PlainTextResponse("RESULTS=000")
+
+        self.logger.info(f"Boardauth request from {sn_decrypt} ({client_ip}) for {req.gameVer}")
 
         resp = MuchaAuthResponse(
-            f"{self.config.mucha.hostname}{':' + str(self.config.allnet.port) if self.config.server.is_develop else ''}"
+            f"{self.config.server.hostname}{':' + str(self.config.server.port) if not self.config.server.is_using_proxy else ''}"
         )
 
+        netid = minfo.get('netid_prefix', "ABxN") + sn_decrypt[5:]
+
+        cab = await self.data.arcade.get_machine(netid)
+        if cab:
+            arcade = await self.data.arcade.get_arcade(cab['id'])
+            if not arcade:
+                self.logger.error(f"Failed to get arcade with id {cab['id']}")
+                return PlainTextResponse("RESULTS=000")
+
+            resp.AREA_0 = arcade["region_id"] or AllnetJapanRegionId.AICHI.name
+            resp.AREA_0_EN = arcade["region_id"] or AllnetJapanRegionId.AICHI.name
+            resp.AREA_FULL_0 = arcade["region_id"] or AllnetJapanRegionId.AICHI.name
+            resp.AREA_FULL_0_EN = arcade["region_id"] or AllnetJapanRegionId.AICHI.name
+            
+            resp.AREA_1 = arcade["country"] or cab['country'] or AllnetCountryCode.JAPAN.value
+            resp.AREA_1_EN = arcade["country"] or cab['country'] or AllnetCountryCode.JAPAN.value
+            resp.AREA_FULL_1 = arcade["country"] or cab['country'] or AllnetCountryCode.JAPAN.value
+            resp.AREA_FULL_1_EN = arcade["country"] or cab['country'] or AllnetCountryCode.JAPAN.value
+
+            resp.AREA_2 = arcade["city"] if arcade["city"] else ""
+            resp.AREA_2_EN = arcade["city"] if arcade["city"] else ""
+            resp.AREA_FULL_2 = arcade["city"] if arcade["city"] else ""
+            resp.AREA_FULL_2_EN = arcade["city"] if arcade["city"] else ""
+
+            resp.AREA_3 = ""
+            resp.AREA_3_EN = ""
+            resp.AREA_FULL_3 = ""
+            resp.AREA_FULL_3_EN = ""
+
+            resp.PREFECTURE_ID = arcade['region_id']
+            resp.COUNTRY_CD = arcade['country'] or cab['country'] or AllnetCountryCode.JAPAN.value
+            resp.PLACE_ID = req.placeId if req.placeId else f"{arcade['country'] or cab['country'] or AllnetCountryCode.JAPAN.value}{arcade['id']:04X}"
+            resp.SHOP_NAME = arcade['name']
+            resp.SHOP_NAME_EN = arcade['name']
+            resp.SHOP_NICKNAME = arcade['nickname']
+            resp.SHOP_NICKNAME_EN = arcade['nickname']
+
+        elif self.config.server.allow_unregistered_serials:
+            self.logger.info(f"Allow unknown serial {netid} ({sn_decrypt}) to auth")
+        
+        else:
+            self.logger.warn(f'Auth failed for NetID {netid}')
+            return PlainTextResponse("RESULTS=000")
+
         self.logger.debug(f"Mucha response {vars(resp)}")
 
-        return self.mucha_postprocess(vars(resp))
+        return PlainTextResponse(self.mucha_postprocess(vars(resp)))
 
-    def handle_updatecheck(self, request: Request, _: Dict) -> bytes:
-        req_dict = self.mucha_preprocess(request.content.getvalue())
+    async def handle_updatecheck(self, request: Request) -> bytes:
+        bod = await request.body()
+        req_dict = self.mucha_preprocess(bod)
         client_ip = Utils.get_ip_addr(request)
 
         if req_dict is None:
             self.logger.error(
-                f"Error processing mucha request {request.content.getvalue()}"
+                f"Error processing mucha request {bod}"
             )
-            return b"RESULTS=000"
+            return PlainTextResponse("RESULTS=000")
 
         req = MuchaUpdateRequest(req_dict)
-        self.logger.info(f"Updatecheck request from {client_ip} for {req.gameVer}")        
+        self.logger.info(f"Updatecheck request from {req.serialNum} ({client_ip}) for {req.gameVer}")        
         self.logger.debug(f"Mucha request {vars(req)}")
 
         if req.gameCd not in self.mucha_registry:
             self.logger.warning(f"Unknown gameCd {req.gameCd}")
-            return b"RESULTS=000"
+            return PlainTextResponse("RESULTS=000")
 
-        resp = MuchaUpdateResponse(req.gameVer, f"{self.config.mucha.hostname}{':' + str(self.config.allnet.port) if self.config.server.is_develop else ''}")
+        resp = MuchaUpdateResponse(req.gameVer, f"{self.config.server.hostname}{':' + str(self.config.server.port) if not self.config.server.is_using_proxy else ''}")
 
         self.logger.debug(f"Mucha response {vars(resp)}")
 
-        return self.mucha_postprocess(vars(resp))
+        return PlainTextResponse(self.mucha_postprocess(vars(resp)))
 
-    def handle_dlstate(self, request: Request, _: Dict) -> bytes:
-        req_dict = self.mucha_preprocess(request.content.getvalue())
+    async def handle_dlstate(self, request: Request) -> bytes:
+        bod = await request.body()
+        req_dict = self.mucha_preprocess(bod)
         client_ip = Utils.get_ip_addr(request)
 
         if req_dict is None:
             self.logger.error(
-                f"Error processing mucha request {request.content.getvalue()}"
+                f"Error processing mucha request {bod}"
             )
-            return b""
+            return PlainTextResponse("RESULTS=000")
         
         req = MuchaDownloadStateRequest(req_dict)
-        self.logger.info(f"DownloadState request from {client_ip} for {req.gameCd} -> {req.updateVer}")        
+        self.logger.info(f"DownloadState request from {req.serialNum} ({client_ip}) for {req.gameCd} -> {req.updateVer}")        
         self.logger.debug(f"request {vars(req)}")
-        return b"RESULTS=001"
+        return PlainTextResponse("RESULTS=001")
 
     def mucha_preprocess(self, data: bytes) -> Optional[Dict]:
         try:
@@ -171,7 +229,7 @@ class MuchaAuthResponse:
         self.RESULTS = "001"
         self.AUTH_INTERVAL = "86400"
         self.SERVER_TIME = datetime.strftime(datetime.now(), "%Y%m%d%H%M")
-        self.UTC_SERVER_TIME = datetime.strftime(datetime.now(pytz.UTC), "%Y%m%d%H%M")
+        self.SERVER_TIME_UTC = datetime.strftime(datetime.now(pytz.UTC), "%Y%m%d%H%M")
 
         self.CHARGE_URL = f"https://{mucha_url}/charge/"
         self.FILE_URL = f"https://{mucha_url}/file/"

core/templates/arcade/index.jinja

@@ -0,0 +1,19 @@
+{% extends "core/templates/index.jinja" %}
+{% block content %}
+{% if arcade is defined %}
+<h1>{{ arcade.name }}</h1>
+<h2>PCBs assigned to this arcade <button class="btn btn-success" id="btn_add_cab" onclick="toggle_add_cab_form()">Add</button></h2>
+{% if success is defined and success == 3 %}
+<div style="background-color: #00AA00; padding: 20px; margin-bottom: 10px; width: 15%;">
+Cab added successfully
+</div>
+{% endif %}
+<ul style="font-size: 20px;">
+{% for c in arcade.cabs %}
+<li><a href="/cab/{{ c.id }}">{{ c.serial }} ({{ c.game }})</a>&nbsp;<button class="btn btn-secondary" onclick="prep_edit_form()">Edit</button>&nbsp;<button class="btn-danger btn">Delete</button></li>
+{% endfor %}
+</ul>
+{% else %}
+<h3>Arcade Not Found</h3>
+{% endif %}
+{% endblock content %}

core/templates/gate/create.jinja

@@ -0,0 +1,24 @@
+{% extends "core/templates/index.jinja" %}
+{% block content %}
+<h1>Create User</h1>
+<form id="create" style="max-width: 240px; min-width: 10%;" action="/gate/gate.create" method="post">
+    <div class="form-group row">
+        <label for="access_code">Card Access Code</label><br>
+        <input class="form-control" name="access_code" id="access_code" type="text" placeholder="00000000000000000000" value={{ code }} maxlength="20" readonly>
+    </div>
+    <div class="form-group row">
+        <label for="username">Username</label><br>
+        <input id="username" class="form-control" name="username" type="text" placeholder="username">
+    </div>
+    <div class="form-group row">
+        <label for="email">Email</label><br>
+        <input id="email" class="form-control" name="email" type="email" placeholder="example@example.com">
+    </div>
+    <div class="form-group row">
+        <label for="passwd">Password</label><br>
+        <input id="passwd" class="form-control" name="passwd" type="password" placeholder="password">
+    </div>
+    <p></p>
+    <input id="submit" class="btn btn-primary" style="display: block; margin: 0 auto;" type="submit" value="Create">
+</form>
+{% endblock content %}

core/templates/gate/gate.jinja

@@ -0,0 +1,32 @@
+{% extends "core/templates/index.jinja" %}
+{% block content %}
+<h1>Gate</h1>
+{% include "core/templates/widgets/err_banner.jinja" %}
+<style>
+    /* Chrome, Safari, Edge, Opera */
+    input::-webkit-outer-spin-button,
+    input::-webkit-inner-spin-button {
+        -webkit-appearance: none;
+        margin: 0;
+    }
+
+    /* Firefox */
+    input[type=number] {
+        -moz-appearance: textfield;
+    }
+</style>
+<form id="login" style="max-width: 240px; min-width: 10%;" action="/gate/gate.login" method="post">
+    <div class="form-group row">
+        <label for="access_code">Card Access Code</label><br>
+        <input form="login" class="form-control" name="access_code" id="access_code" type="number" placeholder="00000000000000000000" maxlength="20" required>
+    </div>
+    <div class="form-group row">
+        <label for="passwd">Password</label><br>
+        <input id="passwd" class="form-control" name="passwd" type="password" placeholder="password">
+    </div>
+    <p></p>
+    <input id="submit" class="btn btn-primary" style="display: block; margin: 0 auto;" form="login" type="submit" value="Login">
+</form>
+<h6>*To register for the webui, type in the access code of your card, as shown in a game, and leave the password field blank.</h6>
+<h6>*If you have not registered a card with this server, you cannot create a webui account.</h6>
+{% endblock content %}

core/templates/index.jinja

@@ -0,0 +1,92 @@
+<!DOCTYPE html>
+<html>
+    <head>
+        <title>{{ title }}</title>
+        <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.2.3/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-rbsA2VBKQhggwzxH7pPCaAqO46MgnOM80zW1RWuH61DGLwZJEdK2Kadq2F9CUG65" crossorigin="anonymous">
+        <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.2.3/dist/js/bootstrap.bundle.min.js" integrity="sha384-kenU1KFdBIe4zVF0s0G1M5b4hcpxyD9F7jL+jjXkk+Q2h455rYXK/7HAuoJl+0I4" crossorigin="anonymous"></script>
+        <script src="https://cdn.jsdelivr.net/npm/jquery@3.2.1/dist/jquery.min.js" integrity="sha256-hwg4gsxgFZhOsEEamdOYGBf13FyQuiTwlAQgxVSNgt4=" crossorigin="anonymous"></script>
+        <style>
+            html {
+                background-color: #181a1b !important;
+                margin: 10px;
+            }
+            html {
+                color-scheme: dark !important;
+            }
+            html, body, input, textarea, select, button, dialog {
+                background-color: #181a1b;
+            }
+            html, body, input, textarea, select, button {
+                border-color: #736b5e;
+                color: #e8e6e3;
+            }
+            a {
+                color: #3391ff;
+            }
+            table {
+                border-color: #545b5e;
+            }
+            ::placeholder {
+                color: #b2aba1;
+            }
+            input:-webkit-autofill,
+            textarea:-webkit-autofill,
+            select:-webkit-autofill {
+                background-color: #404400 !important;
+                color: #e8e6e3 !important;
+            }
+            ::-webkit-scrollbar {
+                background-color: #202324;
+                color: #aba499;
+            }
+            ::-webkit-scrollbar-thumb {
+                background-color: #454a4d;
+            }
+            ::-webkit-scrollbar-thumb:hover {
+                background-color: #575e62;
+            }
+            ::-webkit-scrollbar-thumb:active {
+                background-color: #484e51;
+            }
+            ::-webkit-scrollbar-corner {
+                background-color: #181a1b;
+            }
+            * {
+                scrollbar-color: #454a4d #202324;
+            }
+            ::selection {
+                background-color: #004daa !important;
+                color: #e8e6e3 !important;
+            }
+            ::-moz-selection {
+                background-color: #004daa !important;
+                color: #e8e6e3 !important;
+            }
+            input[type="text"], input[type="text"]:focus, input[type="password"], input[type="password"]:focus, input[type="email"], input[type="email"]:focus {
+                background-color: #202324 !important;
+                color: #e8e6e3;
+            }
+            form {
+                outline: 1px solid grey; 
+                padding: 20px;
+                padding-top: 10px;
+                padding-bottom: 10px;
+            }
+            .err-banner {
+                background-color: #AA0000;
+                padding: 20px;
+                margin-bottom: 10px;
+                width: 15%;
+            }
+            .modal-content {
+                background-color: #181a1b;
+            }
+        </style>
+    </head>
+    <body>
+    {% include "core/templates/widgets/topbar.jinja" %}
+    {% block content %}
+    <h1>{{ server_name }}</h1>
+    {% endblock content %}
+    </body>
+</html>

core/templates/machine/index.jinja

@@ -0,0 +1,4 @@
+{% extends "core/templates/index.jinja" %}
+{% block content %}
+<h1>Machine Management</h1>
+{% endblock content %}

core/templates/sys/index.jinja

@@ -0,0 +1,69 @@
+{% extends "core/templates/index.jinja" %}
+{% block content %}
+<h1>System Management</h1>
+{% if error is defined %}
+{% include "core/templates/widgets/err_banner.jinja" %}
+{% endif %}
+<div class="row" id="rowForm">
+    {% if "{:08b}".format(sesh.permissions)[6] == "1" %}
+    <div class="col-sm-6" style="max-width: 25%;">
+        <form id="usrLookup" name="usrLookup" action="/sys/lookup.user" class="form-inline">
+            <h3>User Search</h3>
+            <div class="form-group">
+                <label for="usrId">User ID</label>
+                <input type="number" class="form-control" id="usrId" name="usrId">
+            </div>
+            OR
+            <div class="form-group">
+                <label for="usrName">Username</label>
+                <input type="text" class="form-control" id="usrName" name="usrName">
+            </div>
+            OR
+            <div class="form-group">
+                <label for="usrEmail">Email address</label>
+                <input type="email" class="form-control" id="usrEmail" name="usrEmail" aria-describedby="emailHelp">
+            </div>
+            <br />
+            <button type="submit" class="btn btn-primary">Search</button>
+        </form>
+    </div>
+    {% endif %}
+    {% if "{:08b}".format(sesh.permissions)[5] == "1" %}
+    <div class="col-sm-6" style="max-width: 25%;">
+        <form id="shopLookup" name="shopLookup" action="/sys/lookup.shop" class="form-inline">
+            <h3>Shop search</h3>
+            <div class="form-group">
+                <label for="shopId">Shop ID</label>
+                <input type="number" class="form-control" id="shopId" name="shopId">
+            </div>
+            OR
+            <div class="form-group">
+                <label for="serialNum">Serial Number</label>
+                <input type="text" class="form-control" id="serialNum" name="serialNum" maxlength="15">
+            </div>
+            <br />
+            <button type="submit" class="btn btn-primary">Search</button>
+        </form>
+    </div>
+    {% endif %}
+</div>
+<div class="row" id="rowResult" style="margin: 10px;">
+    {% if "{:08b}".format(sesh.permissions)[6] == "1" %}
+    <div id="userSearchResult" class="col-sm-6" style="max-width: 25%;">
+        {% for usr in usrlist %}
+        <a href=/user/{{ usr.id }}><pre>{{ usr.username if usr.username is not none else "<i>No Name Set</i>"}}</pre></a>
+        {% endfor %}
+    </div>
+    {% endif %}
+    {% if "{:08b}".format(sesh.permissions)[5] == "1" %}
+    <div id="shopSearchResult" class="col-sm-6" style="max-width: 25%;">
+        {% for shop in shoplist %}
+        <a href="/shop/{{ shop.id }}"><pre>{{ shop.name if shop.name else "<i>No Name Set</i>"}}</pre></a>
+        {% endfor %}
+    </div>
+    {% endif %}
+</div>
+<div class="row" id="rowAdd">
+
+</div>
+{% endblock content %}