HUGE WebUI overhaul

- Removed ugly styles in header
- Properly implemented Bootstrap's NavBar
- Added username to session

.gitattributes

@@ -1,3 +0,0 @@
-*.csv binary
-*.txt binary
-*.json binary

Dockerfile

@@ -1,6 +1,6 @@
 FROM python:3.9.15-slim-bullseye
 
-RUN apt update && apt install default-libmysqlclient-dev build-essential libtk nodejs npm pkg-config -y
+RUN apt update && apt install default-libmysqlclient-dev build-essential libtk nodejs npm -y
 
 WORKDIR /app
 COPY requirements.txt requirements.txt
@@ -12,10 +12,10 @@ RUN chmod +x entrypoint.sh
 
 COPY index.py index.py
 COPY dbutils.py dbutils.py
-COPY read.py read.py
 ADD core core
 ADD titles titles
-ADD logs logs
+ADD config config
+ADD log log
 ADD cert cert
 
 ENTRYPOINT [ "/app/entrypoint.sh" ]

LICENSE.txt

@@ -1,13 +0,0 @@
-           DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
-                   Version 2, December 2004
- 
-Copyright (C) 2004 Sam Hocevar <sam@hocevar.net>
-
-Everyone is permitted to copy and distribute verbatim or modified
-copies of this license document, and changing it is allowed as long
-as the name is changed.
- 
-           DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
-  TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
-
- 0. You just DO WHAT THE FUCK YOU WANT TO.

changelog.md

@@ -1,124 +1,6 @@
 # Changelog
 Documenting updates to ARTEMiS, to be updated every time the master branch is pushed to.
 
-## 20240318
-### CXB
-+ Fixing handle_data_shop_list_detail_request for Sunrise S1
-
-## 20240302
-### SAO
-+ Fixing new profile creation with right heroes and start VP
-+ Fix to the Unanalyzed Log responses returning the wrong rewards
-+ Documentation revised
-
-## 20240226
-### CXB
-+ Fixing paths for rev.py
-+ Changed encoding for handle_data_item_list_icon_request
-
-## 20240202
-### SAO
-+ Added reader assets and edited the game specific documentation
-
-## 20240118
-### System
-+ Added game version names to the readme
-
-## 20240109
-### System
-+ Removed `ADD config config` from dockerfile [#83](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/83) (Thanks zaphkito!)
-
-### Aimedb
-+ Fixed an error that resulted from trying to scan a banned or locked card
-
-## 20240108
-### System
-+ Change how the underlying system handles URLs
-  + This can now allow for things like version-specific, or even keychip-specific URLs
-  + Specific changes to games are noted below
-+ Fix docker files [#60](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/60) (Thanks Rylie!)
-+ Fix support for python 3.8 - 3.10
-
-### Aimedb
-+ Add support for SegaAuth key in games that support it (for now only Chunithm)
-  + This is a JWT that is sent to games, by Aimedb, that the games send to their game server, to verify that the access code the game is sending to the server was obtained via aimedb.
-  + Requires a base64-encoded secret to be set in the `core.yaml`
-
-### Chunithm
-+ Fix Air support
-+ Add saving for userRecentPlayerList
-+ Add support for SegaAuthKey
-+ Fix a bug arising if a user set their name to be 'true' or 'false'
-+ Add support for Sun+ [#78](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/78) (Thanks EmmyHeart!)
-+ Add `matching` section to `chuni.yaml`
-+ ~~Change `udpHolePunchUri` and `reflectorUri` to be STUN and TURN servers~~ Reverted
-+ Imrpove `GetGameSetting` request handling for different versions
-+ Fix issue where songs would not always return all scores [#92](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/92) (Thanks Kumubou!)
-
-### maimai DX
-+ Fix user charges failing to save
-
-### maimai
-+ Made it functional
-
-### CXB
-+ Improvements to request dispatching
-+ Add support for non-omnimix music lists
-
-
-### IDZ
-+ Fix news urls in accordance with the system change to URLs
-
-### Initial D THE ARCADE
-+ Added support for Initial D THE ARCADE S2
-  + Story mode progress added
-  + Bunta Challenge/Touhou Project modes added
-  + Time Trials added
-  + Leaderboards added, but doesn't refresh sometimes
-  + Theory of Street mode added (with CPUs)
-  + Play Stamp/Timetrial events added
-  + Frontend to download profile added
-  + Importer to import profiles added
-
-### ONGEKI
-+ Now supports HTTPS on a per-version basis
-+ Merg PR [#61](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/61) (Thanks phantomlan!)
-  + Add Ranking Event Support
-  + Add reward list support
-  + Add version segregation to Event Ranking, Tech Challenge, and Music Ranking
-  + Now stores ClientTestmode and ClientSetting data
-+ Fix mission points not adding correctly [#68](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/68) (Thanks phantomlan!)
-+ Fix tech challenge [#70](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/70) (Thanks phantomlan!)
-
-### SAO
-+ Change endpoint in accordance with the system change to URLs
-+ Update request header class to be more accurate
-+ Encrypted requests are now supported
-+ Change to using handler classes instead of raw structs for simplicity
-
-### Wacca
-+ Fix a server error causing a seperate error that casued issues
-+ Add better error printing
-+ Add better request validation
-+ Fix HousingStartV2
-+ Fix Lily's housing/get handler
-
-## 20231107
-### CXB
-+  Hotfix `render_POST` sometimes failing to read the request body on large requests
-
-## 20231106
-### CXB
-+ Hotfix `render_POST` function signature signature
-+ Hotfix `handle_action_addenergy_request` hard failing if `get_energy` returns None
-
-## 20231015
-### maimai DX
-+ Added support for FESTiVAL PLUS
-
-### Card Maker
-+ Added support for maimai DX FESTiVAL PLUS
-
 ## 20230716
 ### General
 + Docker files added (#19)

core/__init__.py

@@ -1,6 +1,6 @@
 from core.config import CoreConfig
-from core.allnet import AllnetServlet, BillingServlet
+from core.allnet import AllnetServlet
-from core.aimedb import AimedbServlette
+from core.aimedb import AimedbFactory
 from core.title import TitleServlet
 from core.utils import Utils
 from core.mucha import MuchaServlet

core/adb_handlers/base.py

@@ -102,7 +102,7 @@ class ADBHeader:
         magic, protocol_ver, cmd, length, status, game_id, store_id, keychip_id = struct.unpack_from("<5H6sI12s", data)
         head = cls(magic, protocol_ver, cmd, length, status, game_id, store_id, keychip_id)
         
-        if head.length > len(data):
+        if head.length != len(data):
             raise ADBHeaderException(f"Length is incorrect! Expect {head.length}, got {len(data)}")
         
         return head

core/adb_handlers/felica.py

@@ -52,7 +52,6 @@ class ADBFelicaLookup2Response(ADBBaseResponse):
         self.access_code = access_code if access_code is not None else "00000000000000000000"
         self.company = CompanyCodes.SEGA
         self.portal_status = PortalRegStatus.NO_REG
-        self.auth_key = [0] * 256
 
     @classmethod
     def from_req(cls, req: ADBHeader, user_id: Union[int, None] = None, access_code: Union[str, None] = None) -> "ADBFelicaLookup2Response":
@@ -77,7 +76,7 @@ class ADBFelicaLookup2Response(ADBBaseResponse):
             access_code = bytes.fromhex(self.access_code),
             portal_status = self.portal_status.value,
             company_code = self.company.value,
-            auth_key = self.auth_key
+            auth_key = [0] * 256 # Unsupported
         ))
 
         self.head.length = HEADER_SIZE + len(resp_struct)

core/adb_handlers/lookup.py

@@ -52,7 +52,6 @@ class ADBLookupExResponse(ADBBaseResponse):
         super().__init__(code, length, status, game_id, store_id, keychip_id)
         self.user_id = user_id if user_id is not None else -1
         self.portal_reg = PortalRegStatus.NO_REG
-        self.auth_key = [0] * 256
 
     @classmethod
     def from_req(cls, req: ADBHeader, user_id: Union[int, None]) -> "ADBLookupExResponse":
@@ -73,7 +72,7 @@ class ADBLookupExResponse(ADBBaseResponse):
         body = resp_struct.build(dict(
             user_id = self.user_id,
             portal_reg = self.portal_reg.value,
-            auth_key = self.auth_key,
+            auth_key = [0] * 256,
             relation1 = -1,
             relation2 = -1
         ))

core/aimedb.py

@@ -1,45 +1,24 @@
+from twisted.internet.protocol import Factory, Protocol
 import logging, coloredlogs
 from Crypto.Cipher import AES
-from typing import Dict, Tuple, Callable, Union, Optional
+import struct
-import asyncio
+from typing import Dict, Tuple, Callable, Union
+from typing_extensions import Final
 from logging.handlers import TimedRotatingFileHandler
 
 from core.config import CoreConfig
-from core.utils import create_sega_auth_key
 from core.data import Data
 from .adb_handlers import *
 
-class AimedbServlette():
+
+class AimedbProtocol(Protocol):
     request_list: Dict[int, Tuple[Callable[[bytes, int], Union[ADBBaseResponse, bytes]], int, str]] = {}
+
     def __init__(self, core_cfg: CoreConfig) -> None:
+        self.logger = logging.getLogger("aimedb")
         self.config = core_cfg
         self.data = Data(core_cfg)
-
+        if core_cfg.aimedb.key == "":
-        self.logger = logging.getLogger("aimedb")
-        if not hasattr(self.logger, "initted"):
-            log_fmt_str = "[%(asctime)s] Aimedb | %(levelname)s | %(message)s"
-            log_fmt = logging.Formatter(log_fmt_str)
-
-            fileHandler = TimedRotatingFileHandler(
-                "{0}/{1}.log".format(self.config.server.log_dir, "aimedb"),
-                when="d",
-                backupCount=10,
-            )
-            fileHandler.setFormatter(log_fmt)
-
-            consoleHandler = logging.StreamHandler()
-            consoleHandler.setFormatter(log_fmt)
-
-            self.logger.addHandler(fileHandler)
-            self.logger.addHandler(consoleHandler)
-
-            self.logger.setLevel(self.config.aimedb.loglevel)
-            coloredlogs.install(
-                level=core_cfg.aimedb.loglevel, logger=self.logger, fmt=log_fmt_str
-            )
-            self.logger.initted = True
-
-        if not core_cfg.aimedb.key:
             self.logger.error("!!!KEY NOT SET!!!")
             exit(1)
 
@@ -60,31 +39,27 @@ class AimedbServlette():
 
         self.register_handler(0x13, 0x14, self.handle_log_ex, 'aime_log_ex')
         self.register_handler(0x64, 0x65, self.handle_hello, 'hello')
+        self.register_handler(0x66, 0, self.handle_goodbye, 'goodbye')
     
     def register_handler(self, cmd: int, resp:int, handler: Callable[[bytes, int], Union[ADBBaseResponse, bytes]], name: str) -> None:
         self.request_list[cmd] = (handler, resp, name)
 
-    def start(self) -> None:
+    def append_padding(self, data: bytes):
-        self.logger.info(f"Start on port {self.config.aimedb.port}")
+        """Appends 0s to the end of the data until it's at the correct size"""
-        addr = self.config.aimedb.listen_address if self.config.aimedb.listen_address else self.config.server.listen_address
+        length = struct.unpack_from("<H", data, 6)
-        asyncio.create_task(asyncio.start_server(self.dataReceived, addr, self.config.aimedb.port))
+        padding_size = length[0] - len(data)
+        data += bytes(padding_size)
+        return data
 
-    async def dataReceived(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
+    def connectionMade(self) -> None:
-        self.logger.debug(f"Connection made from {writer.get_extra_info('peername')[0]}")
+        self.logger.debug(f"{self.transport.getPeer().host} Connected")
-        while True:
-            try:
-                data: bytes = await reader.read(4096)
-                if len(data) == 0:
-                    self.logger.debug("Connection closed")
-                    return
-                await self.process_data(data, reader, writer)
-                await writer.drain()
-            except ConnectionResetError as e:
-                self.logger.debug("Connection reset, disconnecting")
-                return
 
-    async def process_data(self, data: bytes, reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> Optional[bytes]:
+    def connectionLost(self, reason) -> None:
-        addr = writer.get_extra_info('peername')[0]
+        self.logger.debug(
+            f"{self.transport.getPeer().host} Disconnected - {reason.value}"
+        )
+
+    def dataReceived(self, data: bytes) -> None:
         cipher = AES.new(self.config.aimedb.key.encode(), AES.MODE_ECB)
 
         try:
@@ -92,9 +67,9 @@ class AimedbServlette():
         
         except Exception as e:
             self.logger.error(f"Failed to decrypt {data.hex()} because {e}")
-            return
+            return None
 
-        self.logger.debug(f"{addr} wrote {decrypted.hex()}")
+        self.logger.debug(f"{self.transport.getPeer().host} wrote {decrypted.hex()}")
 
         try:
             head = ADBHeader.from_data(decrypted)
@@ -103,9 +78,7 @@ class AimedbServlette():
             self.logger.error(f"Error parsing ADB header: {e}")
             try:    
                 encrypted = cipher.encrypt(ADBBaseResponse().make())
-                writer.write(encrypted)
+                self.transport.write(encrypted)
-                await writer.drain()
-                return
 
             except Exception as e:
                 self.logger.error(f"Failed to encrypt default response because {e}")
@@ -115,51 +88,46 @@ class AimedbServlette():
         if head.keychip_id == "ABCD1234567" or head.store_id == 0xfff0:
             self.logger.warning(f"Request from uninitialized AMLib: {vars(head)}")
 
-        if head.cmd == 0x66:
-            self.logger.info("Goodbye")
-            writer.close()
-            return
-
         handler, resp_code, name = self.request_list.get(head.cmd, (self.handle_default, None, 'default'))
 
         if resp_code is None:
             self.logger.warning(f"No handler for cmd {hex(head.cmd)}")
         
         elif resp_code > 0:
-            self.logger.info(f"{name} from {head.keychip_id} ({head.game_id}) @ {addr}")
+            self.logger.info(f"{name} from {head.keychip_id} ({head.game_id}) @ {self.transport.getPeer().host}")
         
-        resp = await handler(decrypted, resp_code)
+        resp = handler(decrypted, resp_code)
 
         if type(resp) == ADBBaseResponse or issubclass(type(resp), ADBBaseResponse):
             resp_bytes = resp.make()
+            if len(resp_bytes) != resp.head.length:
+                resp_bytes = self.append_padding(resp_bytes)
 
         elif type(resp) == bytes:
             resp_bytes = resp
         
         elif resp is None: # Nothing to send, probably a goodbye
-            self.logger.warn(f"None return by handler for {name}")
             return
         
         else:
-            self.logger.error(f"Unsupported type returned by ADB handler for {name}: {type(resp)}")
             raise TypeError(f"Unsupported type returned by ADB handler for {name}: {type(resp)}")
 
         try:    
             encrypted = cipher.encrypt(resp_bytes)
             self.logger.debug(f"Response {resp_bytes.hex()}")
-            writer.write(encrypted)
+            self.transport.write(encrypted)
 
         except Exception as e:
             self.logger.error(f"Failed to encrypt {resp_bytes.hex()} because {e}")
         
-    async def handle_default(self, data: bytes, resp_code: int, length: int = 0x20) -> ADBBaseResponse:
+    def handle_default(self, data: bytes, resp_code: int, length: int = 0x20) -> ADBBaseResponse:
         req = ADBHeader.from_data(data)
         return ADBBaseResponse(resp_code, length, 1, req.game_id, req.store_id, req.keychip_id, req.protocol_ver)
 
-    async def handle_hello(self, data: bytes, resp_code: int) -> ADBBaseResponse:
+    def handle_hello(self, data: bytes, resp_code: int) -> ADBBaseResponse:
-        return await self.handle_default(data, resp_code)
+        return self.handle_default(data, resp_code)
 
-    async def handle_campaign(self, data: bytes, resp_code: int) -> ADBBaseResponse:
+    def handle_campaign(self, data: bytes, resp_code: int) -> ADBBaseResponse:
         h = ADBHeader.from_data(data)
         if h.protocol_ver >= 0x3030:
             req = h
@@ -174,60 +142,29 @@ class AimedbServlette():
         # We don't currently support campaigns
         return resp
 
-    async def handle_lookup(self, data: bytes, resp_code: int) -> ADBBaseResponse:
+    def handle_lookup(self, data: bytes, resp_code: int) -> ADBBaseResponse:
         req = ADBLookupRequest(data)
-        user_id = await self.data.card.get_user_id_from_card(req.access_code)
+        user_id = self.data.card.get_user_id_from_card(req.access_code)
-        is_banned = await self.data.card.get_card_banned(req.access_code)
-        is_locked = await self.data.card.get_card_locked(req.access_code)
 
         ret = ADBLookupResponse.from_req(req.head, user_id)
-        if is_banned and is_locked:
-            ret.head.status = ADBStatus.BAN_SYS_USER
-        elif is_banned:
-            ret.head.status = ADBStatus.BAN_SYS
-        elif is_locked:
-            ret.head.status = ADBStatus.LOCK_USER
         
         self.logger.info(
             f"access_code {req.access_code} -> user_id {ret.user_id}"
         )
-        
-        if user_id and user_id > 0:
-            await self.data.card.update_card_last_login(req.access_code)
         return ret
 
-    async def handle_lookup_ex(self, data: bytes, resp_code: int) -> ADBBaseResponse:
+    def handle_lookup_ex(self, data: bytes, resp_code: int) -> ADBBaseResponse:
         req = ADBLookupRequest(data)
-        user_id = await self.data.card.get_user_id_from_card(req.access_code)
+        user_id = self.data.card.get_user_id_from_card(req.access_code)
-
-        is_banned = await self.data.card.get_card_banned(req.access_code)
-        is_locked = await self.data.card.get_card_locked(req.access_code)
 
         ret = ADBLookupExResponse.from_req(req.head, user_id)
-        if is_banned and is_locked:
-            ret.head.status = ADBStatus.BAN_SYS_USER
-        elif is_banned:
-            ret.head.status = ADBStatus.BAN_SYS
-        elif is_locked:
-            ret.head.status = ADBStatus.LOCK_USER
         
         self.logger.info(
             f"access_code {req.access_code} -> user_id {ret.user_id}"
         )
-
-        if user_id and user_id > 0 and self.config.aimedb.id_secret:
-            auth_key = create_sega_auth_key(user_id, req.head.game_id, req.head.store_id, req.head.keychip_id, self.config.aimedb.id_secret, self.config.aimedb.id_lifetime_seconds)
-            if auth_key is not None:
-                auth_key_extra_len = 256 - len(auth_key)
-                auth_key_full = auth_key.encode() + (b"\0" * auth_key_extra_len)
-                self.logger.debug(f"Generated auth token {auth_key}")
-                ret.auth_key = auth_key_full
-
-        if user_id and user_id > 0:
-            await self.data.card.update_card_last_login(req.access_code)
         return ret
 
-    async def handle_felica_lookup(self, data: bytes, resp_code: int) -> bytes:
+    def handle_felica_lookup(self, data: bytes, resp_code: int) -> bytes:
         """
         On official, I think a card has to be registered for this to actually work, but 
         I'm making the executive decision to not implement that and just kick back our
@@ -243,7 +180,7 @@ class AimedbServlette():
         )
         return ADBFelicaLookupResponse.from_req(req.head, ac)
 
-    async def handle_felica_register(self, data: bytes, resp_code: int) -> bytes:
+    def handle_felica_register(self, data: bytes, resp_code: int) -> bytes:
         """
         I've never seen this used.
         """
@@ -251,14 +188,14 @@ class AimedbServlette():
         ac = self.data.card.to_access_code(req.idm)
         
         if self.config.server.allow_user_registration:
-            user_id = await self.data.user.create_user()
+            user_id = self.data.user.create_user()
 
             if user_id is None:
                 self.logger.error("Failed to register user!")
                 user_id = -1
 
             else:
-                card_id = await self.data.card.create_card(user_id, ac)
+                card_id = self.data.card.create_card(user_id, ac)
 
                 if card_id is None:
                     self.logger.error("Failed to register card!")
@@ -273,14 +210,12 @@ class AimedbServlette():
                 f"Registration blocked!: access code {ac} (IDm: {req.idm} PMm: {req.pmm})"
             )
 
-        if user_id > 0:
-            await self.data.card.update_card_last_login(ac)
         return ADBFelicaLookupResponse.from_req(req.head, ac)
 
-    async def handle_felica_lookup_ex(self, data: bytes, resp_code: int) -> bytes:
+    def handle_felica_lookup_ex(self, data: bytes, resp_code: int) -> bytes:
         req = ADBFelicaLookup2Request(data)
         access_code = self.data.card.to_access_code(req.idm)
-        user_id = await self.data.card.get_user_id_from_card(access_code=access_code)
+        user_id = self.data.card.get_user_id_from_card(access_code=access_code)
 
         if user_id is None:
             user_id = -1
@@ -289,21 +224,9 @@ class AimedbServlette():
             f"idm {req.idm} ipm {req.pmm} -> access_code {access_code} user_id {user_id}"
         )
 
-        resp = ADBFelicaLookup2Response.from_req(req.head, user_id, access_code)
+        return ADBFelicaLookup2Response.from_req(req.head, user_id, access_code)
 
-        if user_id and user_id > 0 and self.config.aimedb.id_secret:
+    def handle_campaign_clear(self, data: bytes, resp_code: int) -> ADBBaseResponse:
-            auth_key = create_sega_auth_key(user_id, req.head.game_id, req.head.store_id, req.head.keychip_id, self.config.aimedb.id_secret, self.config.aimedb.id_lifetime_seconds)
-            if auth_key is not None:
-                auth_key_extra_len = 256 - len(auth_key)
-                auth_key_full = auth_key.encode() + (b"\0" * auth_key_extra_len)
-                self.logger.debug(f"Generated auth token {auth_key}")
-                resp.auth_key = auth_key_full
-        
-        if user_id and user_id > 0:
-            await self.data.card.update_card_last_login(access_code)
-        return resp
-
-    async def handle_campaign_clear(self, data: bytes, resp_code: int) -> ADBBaseResponse:
         req = ADBCampaignClearRequest(data)
 
         resp = ADBCampaignClearResponse.from_req(req.head)
@@ -311,19 +234,19 @@ class AimedbServlette():
         # We don't support campaign stuff
         return resp
 
-    async def handle_register(self, data: bytes, resp_code: int) -> bytes:
+    def handle_register(self, data: bytes, resp_code: int) -> bytes:
         req = ADBLookupRequest(data)
         user_id = -1
         
         if self.config.server.allow_user_registration:
-            user_id = await self.data.user.create_user()
+            user_id = self.data.user.create_user()
 
             if user_id is None:
                 self.logger.error("Failed to register user!")
                 user_id = -1
 
             else:
-                card_id = await self.data.card.create_card(user_id, req.access_code)
+                card_id = self.data.card.create_card(user_id, req.access_code)
 
                 if card_id is None:
                     self.logger.error("Failed to register card!")
@@ -342,23 +265,20 @@ class AimedbServlette():
         if resp.user_id <= 0:
             resp.head.status = ADBStatus.BAN_SYS # Closest we can get to a "You cannot register"
 
-        else:
-            await self.data.card.update_card_last_login(req.access_code)
-
         return resp
 
     # TODO: Save these in some capacity, as deemed relevant
-    async def handle_status_log(self, data: bytes, resp_code: int) -> bytes:
+    def handle_status_log(self, data: bytes, resp_code: int) -> bytes:
         req = ADBStatusLogRequest(data)
         self.logger.info(f"User {req.aime_id} logged {req.status.name} event")
         return ADBBaseResponse(resp_code, 0x20, 1, req.head.game_id, req.head.store_id, req.head.keychip_id, req.head.protocol_ver)
 
-    async def handle_log(self, data: bytes, resp_code: int) -> bytes:
+    def handle_log(self, data: bytes, resp_code: int) -> bytes:
         req = ADBLogRequest(data)
         self.logger.info(f"User {req.aime_id} logged {req.status.name} event, credit_ct: {req.credit_ct} bet_ct: {req.bet_ct} won_ct: {req.won_ct}")
         return ADBBaseResponse(resp_code, 0x20, 1, req.head.game_id, req.head.store_id, req.head.keychip_id, req.head.protocol_ver)
 
-    async def handle_log_ex(self, data: bytes, resp_code: int) -> bytes:
+    def handle_log_ex(self, data: bytes, resp_code: int) -> bytes:
         req = ADBLogExRequest(data)
         strs = []
         self.logger.info(f"Recieved {req.num_logs} or {len(req.logs)} logs")
@@ -367,3 +287,43 @@ class AimedbServlette():
             self.logger.debug(f"User {req.logs[x].aime_id} logged {req.logs[x].status.name} event, credit_ct: {req.logs[x].credit_ct} bet_ct: {req.logs[x].bet_ct} won_ct: {req.logs[x].won_ct}")
         return ADBLogExResponse.from_req(req.head)
 
+    def handle_goodbye(self, data: bytes, resp_code: int) -> None:
+        self.logger.info(f"goodbye from {self.transport.getPeer().host}")
+        self.transport.loseConnection()
+        return
+
+class AimedbFactory(Factory):
+    protocol = AimedbProtocol
+
+    def __init__(self, cfg: CoreConfig) -> None:
+        self.config = cfg
+        log_fmt_str = "[%(asctime)s] Aimedb | %(levelname)s | %(message)s"
+        log_fmt = logging.Formatter(log_fmt_str)
+        self.logger = logging.getLogger("aimedb")
+
+        fileHandler = TimedRotatingFileHandler(
+            "{0}/{1}.log".format(self.config.server.log_dir, "aimedb"),
+            when="d",
+            backupCount=10,
+        )
+        fileHandler.setFormatter(log_fmt)
+
+        consoleHandler = logging.StreamHandler()
+        consoleHandler.setFormatter(log_fmt)
+
+        self.logger.addHandler(fileHandler)
+        self.logger.addHandler(consoleHandler)
+
+        self.logger.setLevel(self.config.aimedb.loglevel)
+        coloredlogs.install(
+            level=cfg.aimedb.loglevel, logger=self.logger, fmt=log_fmt_str
+        )
+
+        if self.config.aimedb.key == "":
+            self.logger.error("Please set 'key' field in your config file.")
+            exit(1)
+
+        self.logger.info(f"Ready on port {self.config.aimedb.port}")
+
+    def buildProtocol(self, addr):
+        return AimedbProtocol(self.config)

core/allnet.py

@@ -1,30 +1,25 @@
+from typing import Dict, List, Any, Optional, Tuple, Union, Final
+import logging, coloredlogs
+from logging.handlers import TimedRotatingFileHandler
+from twisted.web.http import Request
+from datetime import datetime
 import pytz
 import base64
 import zlib
 import json
-import yaml
-import logging
-import coloredlogs
-import urllib.parse
-import math
-from typing import Dict, List, Any, Optional, Union, Final
-from logging.handlers import TimedRotatingFileHandler
-from starlette.requests import Request
-from starlette.responses import PlainTextResponse
-from starlette.applications import Starlette
-from starlette.routing import Route
-from datetime import datetime
 from enum import Enum
 from Crypto.PublicKey import RSA
 from Crypto.Hash import SHA
 from Crypto.Signature import PKCS1_v1_5
-from os import path, environ, mkdir, access, W_OK
+from time import strptime
+from os import path
+import urllib.parse
+import math
 
-from .config import CoreConfig
+from core.config import CoreConfig
-from .utils import Utils
+from core.utils import Utils
-from .data import Data
+from core.data import Data
-from .const import *
+from core.const import *
-from .title import TitleServlet
 
 BILLING_DT_FORMAT: Final[str] = "%Y%m%d%H%M%S"
 
@@ -38,67 +33,13 @@ class ALLNET_STAT(Enum):
     bad_machine = -2
     bad_shop = -3
 
-class DLI_STATUS(Enum):
-    START = 0
-    GET_DOWNLOAD_CONFIGURATION = 1
-    WAIT_DOWNLOAD = 2
-    DOWNLOADING = 3
-
-    NOT_SPECIFY_DLI = 100
-    ONLY_POST_REPORT = 101
-    STOPPED_BY_APP_RELEASE = 102
-    STOPPED_BY_OPT_RELEASE = 103
-
-    DOWNLOAD_COMPLETE_RECENTLY = 110
-
-    DOWNLOAD_COMPLETE_WAIT_RELEASE_TIME = 120
-    DOWNLOAD_COMPLETE_BUT_NOT_SYNC_SERVER = 121
-    DOWNLOAD_COMPLETE_BUT_NOT_FIRST_RESUME = 122
-    DOWNLOAD_COMPLETE_BUT_NOT_FIRST_LAUNCH = 123
-    DOWNLOAD_COMPLETE_WAIT_UPDATE = 124
-
-    DOWNLOAD_COMPLETE_AND_ALREADY_UPDATE = 130
-
-    ERROR_AUTH_FAILURE = 200
-
-    ERROR_GET_DLI_HTTP = 300
-    ERROR_GET_DLI = 301
-    ERROR_PARSE_DLI = 302
-    ERROR_INVALID_GAME_ID = 303
-    ERROR_INVALID_IMAGE_LIST = 304
-    ERROR_GET_DLI_APP = 305
-
-    ERROR_GET_BOOT_ID = 400
-    ERROR_ACCESS_SERVER = 401
-    ERROR_NO_IMAGE = 402
-    ERROR_ACCESS_IMAGE = 403
-
-    ERROR_DOWNLOAD_APP = 500
-    ERROR_DOWNLOAD_OPT = 501
-
-    ERROR_DISK_FULL = 600
-    ERROR_UNINSTALL = 601
-    ERROR_INSTALL_APP = 602
-    ERROR_INSTALL_OPT = 603
-
-    ERROR_GET_DLI_INTERNAL = 900
-    ERROR_ICF = 901
-    ERROR_CHECK_RELEASE_INTERNAL = 902
-    UNKNOWN = 999 # Not the actual enum val but it needs to be here as a catch-all
-
-    @classmethod
-    def from_int(cls, num: int) -> "DLI_STATUS":
-        try:
-            return cls(num)
-        except ValueError:
-            return cls.UNKNOWN
-
 class AllnetServlet:
-    allnet_registry: Dict[str, Any] = {}
     def __init__(self, core_cfg: CoreConfig, cfg_folder: str):
+        super().__init__()
         self.config = core_cfg
         self.config_folder = cfg_folder
         self.data = Data(core_cfg)
+        self.uri_registry: Dict[str, Tuple[str, str]] = {}
 
         self.logger = logging.getLogger("allnet")
         if not hasattr(self.logger, "initialized"):
@@ -124,22 +65,35 @@ class AllnetServlet:
             )
             self.logger.initialized = True
 
-    def startup(self) -> None:
+        plugins = Utils.get_all_titles()
-        self.logger.info(f"Ready on port {self.config.allnet.port if self.config.allnet.standalone else self.config.server.port}")
-        if not TitleServlet.title_registry:
-            TitleServlet(self.config, self.config_folder)
 
-    async def handle_poweron(self, request: Request):
+        if len(plugins) == 0:
+            self.logger.error("No games detected!")
+
+        for _, mod in plugins.items():
+            if hasattr(mod, "index") and hasattr(mod.index, "get_allnet_info"):
+                for code in mod.game_codes:
+                    enabled, uri, host = mod.index.get_allnet_info(
+                        code, self.config, self.config_folder
+                    )
+
+                    if enabled:
+                        self.uri_registry[code] = (uri, host)
+
+        self.logger.info(
+            f"Serving {len(self.uri_registry)} game codes port {core_cfg.allnet.port}"
+        )
+
+    def handle_poweron(self, request: Request, _: Dict):
         request_ip = Utils.get_ip_addr(request)
-        pragma_header = request.headers.get('Pragma', "")
+        pragma_header = request.getHeader('Pragma')
         is_dfi = pragma_header is not None and pragma_header == "DFI"
-        data = await request.body()
         
         try:
             if is_dfi:
-                req_urlencode = self.from_dfi(data)
+                req_urlencode = self.from_dfi(request.content.getvalue())
             else:
-                req_urlencode = data
+                req_urlencode = request.content.getvalue().decode()
 
             req_dict = self.allnet_req_to_dict(req_urlencode)
             if req_dict is None:
@@ -156,7 +110,7 @@ class AllnetServlet:
         except AllnetRequestException as e:
             if e.message != "":
                 self.logger.error(e)
-            return PlainTextResponse()
+            return b""
 
         if req.format_ver == 3:
             resp = AllnetPowerOnResponse3(req.token)
@@ -167,42 +121,42 @@ class AllnetServlet:
 
         self.logger.debug(f"Allnet request: {vars(req)}")
 
-        machine = await self.data.arcade.get_machine(req.serial)        
+        machine = self.data.arcade.get_machine(req.serial)        
         if machine is None and not self.config.server.allow_unregistered_serials:
             msg = f"Unrecognised serial {req.serial} attempted allnet auth from {request_ip}."
-            await self.data.base.log_event(
+            self.data.base.log_event(
                 "allnet", "ALLNET_AUTH_UNKNOWN_SERIAL", logging.WARN, msg
             )
             self.logger.warning(msg)
 
             resp.stat = ALLNET_STAT.bad_machine.value
             resp_dict = {k: v for k, v in vars(resp).items() if v is not None}
-            return PlainTextResponse(urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n")
+            return (urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n").encode("utf-8")
 
         if machine is not None:
-            arcade = await self.data.arcade.get_arcade(machine["arcade"])
+            arcade = self.data.arcade.get_arcade(machine["arcade"])
             if self.config.server.check_arcade_ip:
                 if arcade["ip"] and arcade["ip"] is not None and arcade["ip"] != req.ip:
                     msg = f"Serial {req.serial} attempted allnet auth from bad IP {req.ip} (expected {arcade['ip']})."
-                    await self.data.base.log_event(
+                    self.data.base.log_event(
                         "allnet", "ALLNET_AUTH_BAD_IP", logging.ERROR, msg
                     )
                     self.logger.warning(msg)
 
                     resp.stat = ALLNET_STAT.bad_shop.value
                     resp_dict = {k: v for k, v in vars(resp).items() if v is not None}
-                    return PlainTextResponse(urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n")
+                    return (urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n").encode("utf-8")
                 
-                elif (not arcade["ip"] or arcade["ip"] is None) and self.config.server.strict_ip_checking:
+                elif not arcade["ip"] or arcade["ip"] is None and self.config.server.strict_ip_checking:
                     msg = f"Serial {req.serial} attempted allnet auth from bad IP {req.ip}, but arcade {arcade['id']} has no IP set! (strict checking enabled)."
-                    await self.data.base.log_event(
+                    self.data.base.log_event(
                         "allnet", "ALLNET_AUTH_NO_SHOP_IP", logging.ERROR, msg
                     )
                     self.logger.warning(msg)
 
                     resp.stat = ALLNET_STAT.bad_shop.value
                     resp_dict = {k: v for k, v in vars(resp).items() if v is not None}
-                    return PlainTextResponse(urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n")
+                    return (urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n").encode("utf-8")
 
 
             country = (
@@ -212,7 +166,7 @@ class AllnetServlet:
                 country = AllnetCountryCode.JAPAN.value
 
             resp.country = country
-            resp.place_id = f"{arcade['id']:04X}"
+            resp.place_id = arcade["id"]
             resp.allnet_id = machine["id"]
             resp.name = arcade["name"] if arcade["name"] is not None else ""
             resp.nickname = arcade["nickname"] if arcade["nickname"] is not None else ""
@@ -222,57 +176,53 @@ class AllnetServlet:
                 else AllnetJapanRegionId.AICHI.value
             )
             resp.region_name0 = (
-                arcade["state"]
-                if arcade["state"] is not None
-                else AllnetJapanRegionId.AICHI.name
-            )
-            resp.region_name1 = (
                 arcade["country"]
                 if arcade["country"] is not None
                 else AllnetCountryCode.JAPAN.value
             )
+            resp.region_name1 = (
+                arcade["state"]
+                if arcade["state"] is not None
+                else AllnetJapanRegionId.AICHI.name
+            )
             resp.region_name2 = arcade["city"] if arcade["city"] is not None else ""
-            resp.client_timezone = ( # lmao
+            resp.client_timezone = (
-                arcade["timezone"] if arcade["timezone"] is not None else "+0900" if req.format_ver == 3 else "+09:00"
+                arcade["timezone"] if arcade["timezone"] is not None else "+0900"
             )
         
-        if req.game_id not in TitleServlet.title_registry:
+        if req.game_id not in self.uri_registry:
             if not self.config.server.is_develop:
                 msg = f"Unrecognised game {req.game_id} attempted allnet auth from {request_ip}."
-                await self.data.base.log_event(
+                self.data.base.log_event(
                     "allnet", "ALLNET_AUTH_UNKNOWN_GAME", logging.WARN, msg
                 )
                 self.logger.warning(msg)
 
                 resp.stat = ALLNET_STAT.bad_game.value
                 resp_dict = {k: v for k, v in vars(resp).items() if v is not None}
-                return PlainTextResponse(urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n")
+                return (urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n").encode("utf-8")
 
             else:
                 self.logger.info(
                     f"Allowed unknown game {req.game_id} v{req.ver} to authenticate from {request_ip} due to 'is_develop' being enabled. S/N: {req.serial}"
                 )
-                resp.uri = f"http://{self.config.server.hostname}:{self.config.server.port}/{req.game_id}/{req.ver.replace('.', '')}/"
+                resp.uri = f"http://{self.config.title.hostname}:{self.config.title.port}/{req.game_id}/{req.ver.replace('.', '')}/"
-                resp.host = f"{self.config.server.hostname}:{self.config.server.port}"
+                resp.host = f"{self.config.title.hostname}:{self.config.title.port}"
                 
                 resp_dict = {k: v for k, v in vars(resp).items() if v is not None}
                 resp_str = urllib.parse.unquote(urllib.parse.urlencode(resp_dict))
                 
                 self.logger.debug(f"Allnet response: {resp_str}")
-                return PlainTextResponse(resp_str + "\n")
+                return (resp_str + "\n").encode("utf-8")
 
+        resp.uri, resp.host = self.uri_registry[req.game_id]
 
         int_ver = req.ver.replace(".", "")
-        try:
+        resp.uri = resp.uri.replace("$v", int_ver)
-            resp.uri, resp.host = TitleServlet.title_registry[req.game_id].get_allnet_info(req.game_id, int(int_ver), req.serial)
+        resp.host = resp.host.replace("$v", int_ver)
-        except Exception as e:
-            self.logger.error(f"Error running get_allnet_info for {req.game_id} - {e}")
-            resp.stat = ALLNET_STAT.bad_game.value
-            resp_dict = {k: v for k, v in vars(resp).items() if v is not None}
-            return PlainTextResponse(urllib.parse.unquote(urllib.parse.urlencode(resp_dict)) + "\n")
 
         msg = f"{req.serial} authenticated from {request_ip}: {req.game_id} v{req.ver}"
-        await self.data.base.log_event("allnet", "ALLNET_AUTH_SUCCESS", logging.INFO, msg)
+        self.data.base.log_event("allnet", "ALLNET_AUTH_SUCCESS", logging.INFO, msg)
         self.logger.info(msg)
 
         resp_dict = {k: v for k, v in vars(resp).items() if v is not None}
@@ -284,19 +234,18 @@ class AllnetServlet:
             request.responseHeaders.addRawHeader('Pragma', 'DFI')
             return self.to_dfi(resp_str)"""
 
-        return PlainTextResponse(resp_str)
+        return resp_str.encode("utf-8")
 
-    async def handle_dlorder(self, request: Request):
+    def handle_dlorder(self, request: Request, _: Dict):
         request_ip = Utils.get_ip_addr(request)
-        pragma_header = request.headers.get('Pragma', "")
+        pragma_header = request.getHeader('Pragma')
         is_dfi = pragma_header is not None and pragma_header == "DFI"
-        data = await request.body()
         
         try:
             if is_dfi:
-                req_urlencode = self.from_dfi(data)
+                req_urlencode = self.from_dfi(request.content.getvalue())
             else:
-                req_urlencode = data.decode()
+                req_urlencode = request.content.getvalue().decode()
 
             req_dict = self.allnet_req_to_dict(req_urlencode)
             if req_dict is None:
@@ -313,7 +262,7 @@ class AllnetServlet:
         except AllnetRequestException as e:
             if e.message != "":
                 self.logger.error(e)
-            return PlainTextResponse()
+            return b""
 
         self.logger.info(
             f"DownloadOrder from {request_ip} -> {req.game_id} v{req.ver} serial {req.serial}"
@@ -324,54 +273,54 @@ class AllnetServlet:
             not self.config.allnet.allow_online_updates
             or not self.config.allnet.update_cfg_folder
         ):
-            return PlainTextResponse(urllib.parse.unquote(urllib.parse.urlencode(vars(resp))) + "\n")
+            return urllib.parse.unquote(urllib.parse.urlencode(vars(resp))) + "\n"
 
         else:  # TODO: Keychip check
             if path.exists(
                 f"{self.config.allnet.update_cfg_folder}/{req.game_id}-{req.ver.replace('.', '')}-app.ini"
             ):
-                resp.uri = f"http://{self.config.server.hostname}:{self.config.server.port}/dl/ini/{req.game_id}-{req.ver.replace('.', '')}-app.ini"
+                resp.uri = f"http://{self.config.title.hostname}:{self.config.title.port}/dl/ini/{req.game_id}-{req.ver.replace('.', '')}-app.ini"
 
             if path.exists(
                 f"{self.config.allnet.update_cfg_folder}/{req.game_id}-{req.ver.replace('.', '')}-opt.ini"
             ):
-                resp.uri += f"|http://{self.config.server.hostname}:{self.config.server.port}/dl/ini/{req.game_id}-{req.ver.replace('.', '')}-opt.ini"
+                resp.uri += f"|http://{self.config.title.hostname}:{self.config.title.port}/dl/ini/{req.game_id}-{req.ver.replace('.', '')}-opt.ini"
 
             self.logger.debug(f"Sending download uri {resp.uri}")
-            await self.data.base.log_event("allnet", "DLORDER_REQ_SUCCESS", logging.INFO, f"{Utils.get_ip_addr(request)} requested DL Order for {req.serial} {req.game_id} v{req.ver}")
+            self.data.base.log_event("allnet", "DLORDER_REQ_SUCCESS", logging.INFO, f"{Utils.get_ip_addr(request)} requested DL Order for {req.serial} {req.game_id} v{req.ver}")
 
             res_str = urllib.parse.unquote(urllib.parse.urlencode(vars(resp))) + "\n"
             """if is_dfi:
                 request.responseHeaders.addRawHeader('Pragma', 'DFI')
                 return self.to_dfi(res_str)"""
 
-            return PlainTextResponse(res_str)
+            return res_str
             
-    async def handle_dlorder_ini(self, request: Request) -> bytes:
-        req_file = request.path_params.get("file", "").replace("%0A", "").replace("\n", "")
 
-        if not req_file:
+    def handle_dlorder_ini(self, request: Request, match: Dict) -> bytes:
-            return PlainTextResponse(status_code=404)
+        if "file" not in match:
+            return b""
+
+        req_file = match["file"].replace("%0A", "")
 
         if path.exists(f"{self.config.allnet.update_cfg_folder}/{req_file}"):
             self.logger.info(f"Request for DL INI file {req_file} from {Utils.get_ip_addr(request)} successful")
-            await self.data.base.log_event("allnet", "DLORDER_INI_SENT", logging.INFO, f"{Utils.get_ip_addr(request)} successfully recieved {req_file}")
+            self.data.base.log_event("allnet", "DLORDER_INI_SENT", logging.INFO, f"{Utils.get_ip_addr(request)} successfully recieved {req_file}")
             
-            return PlainTextResponse(open(
+            return open(
-                f"{self.config.allnet.update_cfg_folder}/{req_file}", "r", encoding="utf-8"
+                f"{self.config.allnet.update_cfg_folder}/{req_file}", "rb"
-            ).read())
+            ).read()
 
         self.logger.info(f"DL INI File {req_file} not found")
-        return PlainTextResponse()
+        return b""
 
-    async def handle_dlorder_report(self, request: Request) -> bytes:
+    def handle_dlorder_report(self, request: Request, match: Dict) -> bytes:
-        req_raw = await request.body()
+        req_raw = request.content.getvalue()
-        client_ip = Utils.get_ip_addr(request)
         try:
             req_dict: Dict = json.loads(req_raw)
         except Exception as e:
             self.logger.warning(f"Failed to parse DL Report: {e}")
-            return PlainTextResponse("NG")
+            return "NG"
         
         dl_data_type = DLIMG_TYPE.app
         dl_data = req_dict.get("appimage", {})
@@ -382,24 +331,18 @@ class AllnetServlet:
         
         if dl_data is None or not dl_data:
             self.logger.warning(f"Failed to parse DL Report: Invalid format - contains neither appimage nor optimage")
-            return PlainTextResponse("NG")
+            return "NG"
 
-        rep = DLReport(dl_data, dl_data_type)
+        dl_report_data = DLReport(dl_data, dl_data_type)
 
-        if not rep.validate():
+        if not dl_report_data.validate():
-            self.logger.warning(f"Failed to parse DL Report: Invalid format - {rep.err}")
+            self.logger.warning(f"Failed to parse DL Report: Invalid format - {dl_report_data.err}")
-            return PlainTextResponse("NG")
+            return "NG"
 
-        msg = f"{rep.serial} @ {client_ip} reported {rep.rep_type.name} download state {rep.rf_state.name} for {rep.gd} v{rep.dav}:"\
+        return "OK"
-              f" {rep.tdsc}/{rep.tsc} segments downloaded for working files {rep.wfl} with {rep.dfl if rep.dfl else 'none'} complete."
 
-        await self.data.base.log_event("allnet", "DL_REPORT", logging.INFO, msg, dl_data)
+    def handle_loaderstaterecorder(self, request: Request, match: Dict) -> bytes:
-        self.logger.info(msg)
+        req_data = request.content.getvalue()
-
-        return PlainTextResponse("OK")
-
-    async def handle_loaderstaterecorder(self, request: Request) -> bytes:
-        req_data = await request.body()
         sections = req_data.decode("utf-8").split("\r\n")
         
         req_dict = dict(urllib.parse.parse_qsl(sections[0]))
@@ -411,17 +354,129 @@ class AllnetServlet:
         ip = Utils.get_ip_addr(request)
 
         if serial is None or num_files_dld is None or num_files_to_dl is None or dl_state is None:
-            return PlainTextResponse("NG")
+            return "NG".encode()
 
         self.logger.info(f"LoaderStateRecorder Request from {ip} {serial}: {num_files_dld}/{num_files_to_dl} Files download (State: {dl_state})")
-        return PlainTextResponse("OK")
+        return "OK".encode()
     
-    async def handle_alive(self, request: Request) -> bytes:
+    def handle_alive(self, request: Request, match: Dict) -> bytes:
-        return PlainTextResponse("OK")
+        return "OK".encode()
 
-    async def handle_naomitest(self, request: Request) -> bytes:
+    def handle_billing_request(self, request: Request, _: Dict):
+        req_raw = request.content.getvalue()
+        
+        if request.getHeader('Content-Type') == "application/octet-stream":
+            req_unzip = zlib.decompressobj(-zlib.MAX_WBITS).decompress(req_raw)
+        else:
+            req_unzip = req_raw
+        
+        req_dict = self.billing_req_to_dict(req_unzip)
+        request_ip = Utils.get_ip_addr(request)
+        
+        if req_dict is None:
+            self.logger.error(f"Failed to parse request {request.content.getvalue()}")
+            return b""
+
+        self.logger.debug(f"request {req_dict}")
+
+        rsa = RSA.import_key(open(self.config.billing.signing_key, "rb").read())
+        signer = PKCS1_v1_5.new(rsa)
+        digest = SHA.new()
+        traces: List[TraceData] = []
+
+        try:
+            for x in range(len(req_dict)):
+                if not req_dict[x]:
+                    continue
+                
+                if x == 0:                    
+                    req = BillingInfo(req_dict[x])
+                    continue
+
+                tmp = TraceData(req_dict[x])
+                if tmp.trace_type == TraceDataType.CHARGE:
+                    tmp = TraceDataCharge(req_dict[x])
+                elif tmp.trace_type == TraceDataType.EVENT:
+                    tmp = TraceDataEvent(req_dict[x])
+                elif tmp.trace_type == TraceDataType.CREDIT:
+                    tmp = TraceDataCredit(req_dict[x])
+                
+                traces.append(tmp)
+
+            kc_serial_bytes = req.keychipid.encode()
+        
+        except KeyError as e:
+            self.logger.error(f"Billing request failed to parse: {e}")
+            return f"result=5&linelimit=&message=field is missing or formatting is incorrect\r\n".encode()
+
+        machine = self.data.arcade.get_machine(req.keychipid)
+        if machine is None and not self.config.server.allow_unregistered_serials:
+            msg = f"Unrecognised serial {req.keychipid} attempted billing checkin from {request_ip} for {req.gameid} v{req.gamever}."
+            self.data.base.log_event(
+                "allnet", "BILLING_CHECKIN_NG_SERIAL", logging.WARN, msg
+            )
+            self.logger.warning(msg)
+
+            return f"result=1&requestno={req.requestno}&message=Keychip Serial bad\r\n".encode()
+
+        msg = (
+            f"Billing checkin from {request_ip}: game {req.gameid} ver {req.gamever} keychip {req.keychipid} playcount "
+            f"{req.playcnt} billing_type {req.billingtype.name} nearfull {req.nearfull} playlimit {req.playlimit}"
+        )
+        self.logger.info(msg)
+        self.data.base.log_event("billing", "BILLING_CHECKIN_OK", logging.INFO, msg)
+        if req.traceleft > 0:
+            self.logger.warn(f"{req.traceleft} unsent tracelogs")
+        kc_playlimit = req.playlimit
+        kc_nearfull = req.nearfull
+
+        while req.playcnt > req.playlimit:
+            kc_playlimit += 1024
+            kc_nearfull += 1024
+
+        playlimit = kc_playlimit
+        nearfull = kc_nearfull + (req.billingtype.value * 0x00010000)
+
+        digest.update(playlimit.to_bytes(4, "little") + kc_serial_bytes)
+        playlimit_sig = signer.sign(digest).hex()
+
+        digest = SHA.new()
+        digest.update(nearfull.to_bytes(4, "little") + kc_serial_bytes)
+        nearfull_sig = signer.sign(digest).hex()
+
+        # TODO: playhistory
+
+        #resp = BillingResponse(playlimit, playlimit_sig, nearfull, nearfull_sig)
+        resp = BillingResponse(playlimit, playlimit_sig, nearfull, nearfull_sig, req.requestno, req.protocolver)
+
+        resp_str = urllib.parse.unquote(urllib.parse.urlencode(vars(resp))) + "\r\n"
+
+        self.logger.debug(f"response {vars(resp)}")
+        if req.traceleft > 0:
+            self.logger.info(f"Requesting 20 more of {req.traceleft} unsent tracelogs")
+            return f"result=6&waittime=0&linelimit=20\r\n".encode()
+
+        return resp_str.encode("utf-8")
+
+    def handle_naomitest(self, request: Request, _: Dict) -> bytes:
         self.logger.info(f"Ping from {Utils.get_ip_addr(request)}")
-        return PlainTextResponse("naomi ok")
+        return b"naomi ok"
+
+    def billing_req_to_dict(self, data: bytes):
+        """
+        Parses an billing request string into a python dictionary
+        """
+        try:
+            sections = data.decode("ascii").split("\r\n")
+
+            ret = []
+            for x in sections:
+                ret.append(dict(urllib.parse.parse_qsl(x)))
+            return ret
+
+        except Exception as e:
+            self.logger.error(f"billing_req_to_dict: {e} while parsing {data}")
+            return None
 
     def allnet_req_to_dict(self, data: str) -> Optional[List[Dict[str, Any]]]:
         """
@@ -449,150 +504,6 @@ class AllnetServlet:
         zipped = zlib.compress(unzipped)
         return base64.b64encode(zipped)
 
-class BillingServlet:
-    def __init__(self, core_cfg: CoreConfig, cfg_folder: str) -> None:
-        self.config = core_cfg
-        self.config_folder = cfg_folder
-        self.data = Data(core_cfg)
-
-        self.logger = logging.getLogger("billing")
-        if not hasattr(self.logger, "initialized"):
-            log_fmt_str = "[%(asctime)s] Billing | %(levelname)s | %(message)s"
-            log_fmt = logging.Formatter(log_fmt_str)
-
-            fileHandler = TimedRotatingFileHandler(
-                "{0}/{1}.log".format(self.config.server.log_dir, "billing"),
-                when="d",
-                backupCount=10,
-            )
-            fileHandler.setFormatter(log_fmt)
-
-            consoleHandler = logging.StreamHandler()
-            consoleHandler.setFormatter(log_fmt)
-
-            self.logger.addHandler(fileHandler)
-            self.logger.addHandler(consoleHandler)
-
-            self.logger.setLevel(core_cfg.allnet.loglevel)
-            coloredlogs.install(
-                level=core_cfg.billing.loglevel, logger=self.logger, fmt=log_fmt_str
-            )
-            self.logger.initialized = True
-    
-    def startup(self) -> None:
-        self.logger.info(f"Ready on port {self.config.billing.port if self.config.billing.standalone else self.config.server.port}")
-
-    def billing_req_to_dict(self, data: bytes):
-        """
-        Parses an billing request string into a python dictionary
-        """
-        try:
-            sections = data.decode("ascii").split("\r\n")
-
-            ret = []
-            for x in sections:
-                ret.append(dict(urllib.parse.parse_qsl(x)))
-            return ret
-
-        except Exception as e:
-            self.logger.error(f"billing_req_to_dict: {e} while parsing {data}")
-            return None
-    
-    async def handle_billing_request(self, request: Request):
-        req_raw = await request.body()
-        
-        if request.headers.get('Content-Type', '') == "application/octet-stream":
-            req_unzip = zlib.decompressobj(-zlib.MAX_WBITS).decompress(req_raw)
-        else:
-            req_unzip = req_raw
-        
-        req_dict = self.billing_req_to_dict(req_unzip)
-        request_ip = Utils.get_ip_addr(request)
-        
-        if req_dict is None:
-            self.logger.error(f"Failed to parse request {req_raw}")
-            return PlainTextResponse()
-
-        self.logger.debug(f"request {req_dict}")
-
-        rsa = RSA.import_key(open(self.config.billing.signing_key, "rb").read())
-        signer = PKCS1_v1_5.new(rsa)
-        digest = SHA.new()
-        traces: List[TraceData] = []
-        try:
-            req = BillingInfo(req_dict[0])
-        except KeyError as e:
-            self.logger.error(f"Billing request failed to parse: {e}")
-            return PlainTextResponse("result=5&linelimit=&message=field is missing or formatting is incorrect\r\n")
-
-        for x in range(1, len(req_dict)):
-            if not req_dict[x]:
-                continue
-            
-            try:
-                tmp = TraceData(req_dict[x])
-                if tmp.trace_type == TraceDataType.CHARGE:
-                    tmp = TraceDataCharge(req_dict[x])
-                elif tmp.trace_type == TraceDataType.EVENT:
-                    tmp = TraceDataEvent(req_dict[x])
-                elif tmp.trace_type == TraceDataType.CREDIT:
-                    tmp = TraceDataCredit(req_dict[x])
-                
-                traces.append(tmp)
-            
-            except KeyError as e:
-                self.logger.warn(f"Tracelog failed to parse: {e}")
-
-        kc_serial_bytes = req.keychipid.encode()
-        
-
-        machine = await self.data.arcade.get_machine(req.keychipid)
-        if machine is None and not self.config.server.allow_unregistered_serials:
-            msg = f"Unrecognised serial {req.keychipid} attempted billing checkin from {request_ip} for {req.gameid} v{req.gamever}."
-            await self.data.base.log_event(
-                "allnet", "BILLING_CHECKIN_NG_SERIAL", logging.WARN, msg
-            )
-            self.logger.warning(msg)
-
-            return PlainTextResponse(f"result=1&requestno={req.requestno}&message=Keychip Serial bad\r\n")
-
-        msg = (
-            f"Billing checkin from {request_ip}: game {req.gameid} ver {req.gamever} keychip {req.keychipid} playcount "
-            f"{req.playcnt} billing_type {req.billingtype.name} nearfull {req.nearfull} playlimit {req.playlimit}"
-        )
-        self.logger.info(msg)
-        await self.data.base.log_event("billing", "BILLING_CHECKIN_OK", logging.INFO, msg)
-        if req.traceleft > 0:
-            self.logger.warn(f"{req.traceleft} unsent tracelogs")
-        kc_playlimit = req.playlimit
-        kc_nearfull = req.nearfull
-
-        while req.playcnt > req.playlimit:
-            kc_playlimit += 1024
-            kc_nearfull += 1024
-
-        playlimit = kc_playlimit
-        nearfull = kc_nearfull + (req.billingtype.value * 0x00010000)
-
-        digest.update(playlimit.to_bytes(4, "little") + kc_serial_bytes)
-        playlimit_sig = signer.sign(digest).hex()
-
-        digest = SHA.new()
-        digest.update(nearfull.to_bytes(4, "little") + kc_serial_bytes)
-        nearfull_sig = signer.sign(digest).hex()
-
-        # TODO: playhistory
-
-        resp = BillingResponse(playlimit, playlimit_sig, nearfull, nearfull_sig, req.requestno, req.protocolver)
-
-        resp_str = urllib.parse.unquote(urllib.parse.urlencode(vars(resp))) + "\r\n"
-
-        self.logger.debug(f"response {vars(resp)}")
-        if req.traceleft > 0:
-            self.logger.info(f"Requesting 20 more of {req.traceleft} unsent tracelogs")
-            return PlainTextResponse("result=6&waittime=0&linelimit=20\r\n")
-        
-        return PlainTextResponse(resp_str)
 
 class AllnetPowerOnRequest:
     def __init__(self, req: Dict) -> None:
@@ -614,7 +525,7 @@ class AllnetPowerOnResponse:
         self.stat = 1
         self.uri = ""
         self.host = ""
-        self.place_id = "0123"
+        self.place_id = "123"
         self.name = "ARTEMiS"
         self.nickname = "ARTEMiS"
         self.region0 = "1"
@@ -652,6 +563,7 @@ class AllnetPowerOnResponse3(AllnetPowerOnResponse):
         self.minute = None
         self.second = None
 
+
 class AllnetPowerOnResponse2(AllnetPowerOnResponse):
     def __init__(self) -> None:
         super().__init__()
@@ -661,6 +573,7 @@ class AllnetPowerOnResponse2(AllnetPowerOnResponse):
         self.timezone = "+09:00"
         self.res_class = "PowerOnResponseV2"
 
+
 class AllnetDownloadOrderRequest:
     def __init__(self, req: Dict) -> None:
         self.game_id = req.get("game_id", "")
@@ -668,6 +581,7 @@ class AllnetDownloadOrderRequest:
         self.serial = req.get("serial", "")
         self.encode = req.get("encode", "")
 
+
 class AllnetDownloadOrderResponse:
     def __init__(self, stat: int = 1, serial: str = "", uri: str = "") -> None:
         self.stat = stat
@@ -730,17 +644,9 @@ class TraceData:
             self.seq_number = int(data.get("sn", None))
             self.trace_type = TraceDataType(int(data.get("tt", None)))
             self.date_sync_flg = bool(data.get("ds", None))
-            
+            self.date = datetime.strptime(data.get("dt", None), BILLING_DT_FORMAT)
-            dt = data.get("dt", None)
-            if dt is None:
-                raise KeyError("dt not present")
-            if dt == "20000000000000": # Not sure what causes it to send like this...
-                self.date = datetime(2000, 1, 1, 0, 0, 0, 0)
-            else:
-                self.date = datetime.strptime(data.get("dt", None), BILLING_DT_FORMAT)
-            
             self.keychip = str(data.get("kn", None))
-            self.lib_ver = float(data.get("alib", 0))
+            self.lib_ver = float(data.get("alib", None))
         except Exception as e:
             raise KeyError(e)
 
@@ -748,8 +654,8 @@ class TraceDataCharge(TraceData):
     def __init__(self, data: Dict) -> None:
         super().__init__(data)
         try:
-            self.game_id = str(data.get("gi", None)) # these seem optional...?
+            self.game_id = str(data.get("gi", None))
-            self.game_version = float(data.get("gv", 0))
+            self.game_version = float(data.get("gv", None))
             self.board_serial = str(data.get("bn", None))
             self.shop_ip = str(data.get("ti", None))
             self.play_count = int(data.get("pc", None))
@@ -817,6 +723,7 @@ class BillingResponse:
         # playhistory -> YYYYMM/C:...
         # YYYY -> 4 digit year, MM -> 2 digit month, C -> Playcount during that period
 
+
 class AllnetRequestException(Exception):
     def __init__(self, message="") -> None:
         self.message = message
@@ -833,13 +740,13 @@ class DLReport:
         self.ot = data.get("ot")
         self.rt = data.get("rt")
         self.as_ = data.get("as")
-        self.rf_state = DLI_STATUS.from_int(data.get("rf_state"))
+        self.rf_state = data.get("rf_state")
         self.gd = data.get("gd")
         self.dav = data.get("dav")
         self.wdav = data.get("wdav") # app only
         self.dov = data.get("dov")
         self.wdov = data.get("wdov") # app only
-        self.rep_type = report_type
+        self.__type = report_type
         self.err = ""
     
     def validate(self) -> bool:
@@ -847,6 +754,14 @@ class DLReport:
             self.err = "serial not provided"
             return False
         
+        if self.dfl is None: 
+            self.err = "dfl not provided"
+            return False
+        
+        if self.wfl is None:
+            self.err = "wfl not provided"
+            return False
+        
         if self.tsc is None:
             self.err = "tsc not provided"
             return False
@@ -855,6 +770,18 @@ class DLReport:
             self.err = "tdsc not provided"
             return False
         
+        if self.at is None:
+            self.err = "at not provided"
+            return False
+        
+        if self.ot is None:
+            self.err = "ot not provided"
+            return False
+        
+        if self.rt is None:
+            self.err = "rt not provided"
+            return False
+        
         if self.as_ is None:
             self.err = "as not provided"
             return False
@@ -875,57 +802,12 @@ class DLReport:
             self.err = "dov not provided"
             return False
         
-        if (self.wdav is None or self.wdov is None) and self.rep_type == DLIMG_TYPE.app:
+        if (self.wdav is None or self.wdov is None) and self.__type == DLIMG_TYPE.app:
             self.err = "wdav or wdov not provided in app image"
             return False
         
-        if (self.wdav is not None or self.wdov is not None) and self.rep_type == DLIMG_TYPE.opt:
+        if (self.wdav is not None or self.wdov is not None) and self.__type == DLIMG_TYPE.opt:
             self.err = "wdav or wdov provided in opt image"
             return False
         
         return True
-
-cfg_dir = environ.get("DIANA_CFG_DIR", "config")
-cfg: CoreConfig = CoreConfig()
-if path.exists(f"{cfg_dir}/core.yaml"):
-    cfg.update(yaml.safe_load(open(f"{cfg_dir}/core.yaml")))
-
-if not path.exists(cfg.server.log_dir):
-    mkdir(cfg.server.log_dir)
-
-if not access(cfg.server.log_dir, W_OK):
-    print(
-        f"Log directory {cfg.server.log_dir} NOT writable, please check permissions"
-    )
-    exit(1)
-
-billing = BillingServlet(cfg, cfg_dir)
-app_billing = Starlette(
-    cfg.server.is_develop, 
-    [        
-        Route("/request", billing.handle_billing_request, methods=["POST"]),
-        Route("/request/", billing.handle_billing_request, methods=["POST"]),
-    ],
-    on_startup=[billing.startup]
-)
-
-allnet = AllnetServlet(cfg, cfg_dir)
-route_lst = [
-    Route("/sys/servlet/PowerOn", allnet.handle_poweron, methods=["GET", "POST"]),
-    Route("/sys/servlet/DownloadOrder", allnet.handle_dlorder, methods=["GET", "POST"]),
-    Route("/sys/servlet/LoaderStateRecorder", allnet.handle_loaderstaterecorder, methods=["GET", "POST"]),
-    Route("/sys/servlet/Alive", allnet.handle_alive, methods=["GET", "POST"]),
-    Route("/naomitest.html", allnet.handle_naomitest),
-]
-
-if cfg.allnet.allow_online_updates:
-    route_lst += [
-        Route("/report-api/Report", allnet.handle_dlorder_report, methods=["POST"]),
-        Route("/dl/ini/{file:str}", allnet.handle_dlorder_ini),
-    ]
-
-app_allnet = Starlette(
-    cfg.server.is_develop, 
-    route_lst,
-    on_startup=[allnet.startup]
-)

core/app.py

@@ -1,92 +0,0 @@
-import yaml
-import logging
-import coloredlogs
-from logging.handlers import TimedRotatingFileHandler
-from starlette.routing import Route
-from starlette.requests import Request
-from starlette.applications import Starlette
-from starlette.responses import PlainTextResponse
-from os import environ, path, mkdir, W_OK, access
-from typing import List
-
-from core import CoreConfig, TitleServlet, MuchaServlet, AllnetServlet, BillingServlet, AimedbServlette
-from core.frontend import FrontendServlet
-
-async def dummy_rt(request: Request):
-    return PlainTextResponse("Service OK")
-
-cfg_dir = environ.get("ARTEMIS_CFG_DIR", "config")
-cfg: CoreConfig = CoreConfig()
-if path.exists(f"{cfg_dir}/core.yaml"):
-    cfg.update(yaml.safe_load(open(f"{cfg_dir}/core.yaml")))
-
-if not path.exists(cfg.server.log_dir):
-    mkdir(cfg.server.log_dir)
-
-if not access(cfg.server.log_dir, W_OK):
-    print(
-        f"Log directory {cfg.server.log_dir} NOT writable, please check permissions"
-    )
-    exit(1)
-
-logger = logging.getLogger("core")
-log_fmt_str = "[%(asctime)s] Core | %(levelname)s | %(message)s"
-log_fmt = logging.Formatter(log_fmt_str)
-
-fileHandler = TimedRotatingFileHandler(
-    "{0}/{1}.log".format(cfg.server.log_dir, "core"), when="d", backupCount=10
-)
-fileHandler.setFormatter(log_fmt)
-
-consoleHandler = logging.StreamHandler()
-consoleHandler.setFormatter(log_fmt)
-
-logger.addHandler(fileHandler)
-logger.addHandler(consoleHandler)
-
-log_lv = logging.DEBUG if cfg.server.is_develop else logging.INFO
-logger.setLevel(log_lv)
-coloredlogs.install(level=log_lv, logger=logger, fmt=log_fmt_str)
-
-logger.info(f"Artemis starting in {'develop' if cfg.server.is_develop else 'production'} mode")
-
-title = TitleServlet(cfg, cfg_dir) # This has to be loaded first to load plugins
-mucha = MuchaServlet(cfg, cfg_dir)
-
-route_lst: List[Route] = [
-    # Mucha
-    Route("/mucha_front/boardauth.do", mucha.handle_boardauth, methods=["POST"]),
-    Route("/mucha_front/updatacheck.do", mucha.handle_updatecheck, methods=["POST"]),
-    Route("/mucha_front/downloadstate.do", mucha.handle_dlstate, methods=["POST"]),
-    # General
-    Route("/", dummy_rt),
-    Route("/robots.txt", FrontendServlet.robots)
-]
-
-if not cfg.billing.standalone:
-    billing = BillingServlet(cfg, cfg_dir)
-    route_lst += [
-        Route("/request", billing.handle_billing_request, methods=["POST"]),
-        Route("/request/", billing.handle_billing_request, methods=["POST"]),
-    ]
-
-if not cfg.allnet.standalone:
-    allnet = AllnetServlet(cfg, cfg_dir)
-    route_lst += [
-        Route("/sys/servlet/PowerOn", allnet.handle_poweron, methods=["GET", "POST"]),
-        Route("/sys/servlet/DownloadOrder", allnet.handle_dlorder, methods=["GET", "POST"]),
-        Route("/sys/servlet/LoaderStateRecorder", allnet.handle_loaderstaterecorder, methods=["GET", "POST"]),
-        Route("/sys/servlet/Alive", allnet.handle_alive, methods=["GET", "POST"]),
-        Route("/naomitest.html", allnet.handle_naomitest),
-    ]
-    
-    if cfg.allnet.allow_online_updates:
-        route_lst += [
-            Route("/report-api/Report", allnet.handle_dlorder_report, methods=["POST"]),
-            Route("/dl/ini/{file:str}", allnet.handle_dlorder_ini),
-        ]
-
-for code, game in title.title_registry.items():
-    route_lst += game.get_routes()
-
-app = Starlette(cfg.server.is_develop, route_lst)

core/config.py

@@ -1,49 +1,17 @@
 import logging, os
 from typing import Any
 
+
 class ServerConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
     @property
     def listen_address(self) -> str:
-        """
-        Address Artemis will bind to and listen on
-        """
         return CoreConfig.get_config_field(
             self.__config, "core", "server", "listen_address", default="127.0.0.1"
         )
 
-    @property
-    def hostname(self) -> str:
-        """
-        Hostname sent to games
-        """
-        return CoreConfig.get_config_field(
-            self.__config, "core", "server", "hostname", default="localhost"
-        )
-    
-    @property
-    def port(self) -> int:
-        """
-        Port the game will listen on
-        """
-        return CoreConfig.get_config_field(
-            self.__config, "core", "server", "port", default=80
-        )
-
-    @property
-    def ssl_key(self) -> str:
-        return CoreConfig.get_config_field(
-            self.__config, "core", "server", "ssl_key", default="cert/title.key"
-        )
-
-    @property
-    def ssl_cert(self) -> str:
-        return CoreConfig.get_config_field(
-            self.__config, "core", "title", "ssl_cert", default="cert/title.pem"
-        )
-
     @property
     def allow_user_registration(self) -> bool:
         return CoreConfig.get_config_field(
@@ -69,29 +37,9 @@ class ServerConfig:
         )
 
     @property
-    def is_using_proxy(self) -> bool:
+    def threading(self) -> bool:
         return CoreConfig.get_config_field(
-            self.__config, "core", "server", "is_using_proxy", default=False
+            self.__config, "core", "server", "threading", default=False
-        )
-
-    @property
-    def proxy_port(self) -> int:
-        """
-        What port the proxy is listening on. This will be sent instead of 'port' if 
-        is_using_proxy is True and this value is non-zero
-        """
-        return CoreConfig.get_config_field(
-            self.__config, "core", "server", "proxy_port", default=0
-        )
-
-    @property
-    def proxy_port_ssl(self) -> int:
-        """
-        What port the proxy is listening for secure connections on. This will be sent 
-        instead of 'port' if is_using_proxy is True and this value is non-zero
-        """
-        return CoreConfig.get_config_field(
-            self.__config, "core", "server", "proxy_port_ssl", default=0
         )
 
     @property
@@ -112,6 +60,7 @@ class ServerConfig:
             self.__config, "core", "server", "strict_ip_checking", default=False
         )
 
+
 class TitleConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
@@ -125,17 +74,18 @@ class TitleConfig:
         )
 
     @property
-    def reboot_start_time(self) -> str:
+    def hostname(self) -> str:
         return CoreConfig.get_config_field(
-            self.__config, "core", "title", "reboot_start_time", default=""
+            self.__config, "core", "title", "hostname", default="localhost"
         )
 
     @property
-    def reboot_end_time(self) -> str:
+    def port(self) -> int:
         return CoreConfig.get_config_field(
-            self.__config, "core", "title", "reboot_end_time", default=""
+            self.__config, "core", "title", "port", default=8080
         )
 
+
 class DatabaseConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
@@ -173,7 +123,7 @@ class DatabaseConfig:
     @property
     def protocol(self) -> str:
         return CoreConfig.get_config_field(
-            self.__config, "core", "database", "protocol", default="mysql"
+            self.__config, "core", "database", "type", default="mysql"
         )
 
     @property
@@ -191,9 +141,13 @@ class DatabaseConfig:
         )
 
     @property
-    def enable_memcached(self) -> bool:
+    def user_table_autoincrement_start(self) -> int:
         return CoreConfig.get_config_field(
-            self.__config, "core", "database", "enable_memcached", default=True
+            self.__config,
+            "core",
+            "database",
+            "user_table_autoincrement_start",
+            default=10000,
         )
 
     @property
@@ -202,12 +156,13 @@ class DatabaseConfig:
             self.__config, "core", "database", "memcached_host", default="localhost"
         )
 
+
 class FrontendConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
     @property
-    def enable(self) -> bool:
+    def enable(self) -> int:
         return CoreConfig.get_config_field(
             self.__config, "core", "frontend", "enable", default=False
         )
@@ -215,7 +170,7 @@ class FrontendConfig:
     @property
     def port(self) -> int:
         return CoreConfig.get_config_field(
-            self.__config, "core", "frontend", "port", default=8080
+            self.__config, "core", "frontend", "port", default=8090
         )
 
     @property
@@ -226,20 +181,17 @@ class FrontendConfig:
             )
         )
 
-    @property
-    def secret(self) -> str:
-        return CoreConfig.get_config_field(
-            self.__config, "core", "frontend", "secret", default=""
-        )
 
 class AllnetConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
     @property
-    def standalone(self) -> bool:
+    def loglevel(self) -> int:
-        return CoreConfig.get_config_field(
+        return CoreConfig.str_to_loglevel(
-            self.__config, "core", "allnet", "standalone", default=False
+            CoreConfig.get_config_field(
+                self.__config, "core", "allnet", "loglevel", default="info"
+            )
         )
 
     @property
@@ -249,11 +201,9 @@ class AllnetConfig:
         )
 
     @property
-    def loglevel(self) -> int:
+    def ip_check(self) -> bool:
-        return CoreConfig.str_to_loglevel(
+        return CoreConfig.get_config_field(
-            CoreConfig.get_config_field(
+            self.__config, "core", "allnet", "ip_check", default=False
-                self.__config, "core", "allnet", "loglevel", default="info"
-            )
         )
 
     @property
@@ -268,24 +218,11 @@ class AllnetConfig:
             self.__config, "core", "allnet", "update_cfg_folder", default=""
         )
 
+
 class BillingConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
-    @property
-    def standalone(self) -> bool:
-        return CoreConfig.get_config_field(
-            self.__config, "core", "billing", "standalone", default=True
-        )
-
-    @property
-    def loglevel(self) -> int:
-        return CoreConfig.str_to_loglevel(
-            CoreConfig.get_config_field(
-                self.__config, "core", "billing", "loglevel", default="info"
-            )
-        )
-
     @property
     def port(self) -> int:
         return CoreConfig.get_config_field(
@@ -310,22 +247,11 @@ class BillingConfig:
             self.__config, "core", "billing", "signing_key", default="cert/billing.key"
         )
 
+
 class AimedbConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
-    @property
-    def enable(self) -> bool:
-        return CoreConfig.get_config_field(
-            self.__config, "core", "aimedb", "enable", default=True
-        )
-
-    @property
-    def listen_address(self) -> bool:
-        return CoreConfig.get_config_field(
-            self.__config, "core", "aimedb", "listen_address", default=""
-        )
-
     @property
     def loglevel(self) -> int:
         return CoreConfig.str_to_loglevel(
@@ -346,22 +272,17 @@ class AimedbConfig:
             self.__config, "core", "aimedb", "key", default=""
         )
 
-    @property
-    def id_secret(self) -> str:
-        return CoreConfig.get_config_field(
-            self.__config, "core", "aimedb", "id_secret", default=""
-        )
-
-    @property
-    def id_lifetime_seconds(self) -> int:
-        return CoreConfig.get_config_field(
-            self.__config, "core", "aimedb", "id_lifetime_seconds", default=86400
-        )
 
 class MuchaConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
+    @property
+    def enable(self) -> int:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "mucha", "enable", default=False
+        )
+
     @property
     def loglevel(self) -> int:
         return CoreConfig.str_to_loglevel(
@@ -370,6 +291,13 @@ class MuchaConfig:
             )
         )
 
+    @property
+    def hostname(self) -> str:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "mucha", "hostname", default="localhost"
+        )
+
+
 class CoreConfig(dict):
     def __init__(self) -> None:
         self.server = ServerConfig(self)
@@ -392,19 +320,6 @@ class CoreConfig(dict):
         else:
             return logging.INFO
 
-    @classmethod
-    def loglevel_to_str(cls, level: int) -> str:
-        if level == logging.ERROR:
-            return "error"
-        elif level == logging.WARN:
-            return "warn"
-        elif level == logging.INFO:
-            return "info"
-        elif level == logging.DEBUG:
-            return "debug"
-        else:
-            return "notset"
-
     @classmethod
     def get_config_field(
         cls, __config: dict, module, *path: str, default: Any = ""

core/data/alembic/README

@@ -1 +0,0 @@
-Generic single-database configuration.

core/data/alembic/alembic.ini

@@ -1,64 +0,0 @@
-# A generic, single database configuration.
-
-[alembic]
-script_location=.
-
-# template used to generate migration files
-# file_template = %%(rev)s_%%(slug)s
-
-# max length of characters to apply to the
-# "slug" field
-#truncate_slug_length = 40
-
-# set to 'true' to run the environment during
-# the 'revision' command, regardless of autogenerate
-# revision_environment = false
-
-# set to 'true' to allow .pyc and .pyo files without
-# a source .py file to be detected as revisions in the
-# versions/ directory
-# sourceless = false
-
-# version location specification; this defaults
-# to migrations//versions.  When using multiple version
-# directories, initial revisions must be specified with --version-path
-# version_locations = %(here)s/bar %(here)s/bat migrations//versions
-
-# the output encoding used when revision files
-# are written from script.py.mako
-# output_encoding = utf-8
-
-# Logging configuration
-[loggers]
-keys = root,sqlalchemy,alembic
-
-[handlers]
-keys = console
-
-[formatters]
-keys = generic
-
-[logger_root]
-level = WARN
-handlers = console
-qualname =
-
-[logger_sqlalchemy]
-level = WARN
-handlers =
-qualname = sqlalchemy.engine
-
-[logger_alembic]
-level = INFO
-handlers =
-qualname = alembic
-
-[handler_console]
-class = StreamHandler
-args = (sys.stderr,)
-level = NOTSET
-formatter = generic
-
-[formatter_generic]
-format = %(levelname)-5.5s [%(name)s] %(message)s
-datefmt = %H:%M:%S

core/data/alembic/env.py

@@ -1,81 +0,0 @@
-from __future__ import with_statement
-from alembic import context
-from sqlalchemy import engine_from_config, pool
-from logging.config import fileConfig
-
-from core.data.schema.base import metadata
-
-# this is the Alembic Config object, which provides
-# access to the values within the .ini file in use.
-config = context.config
-
-# Interpret the config file for Python logging.
-# This line sets up loggers basically.
-fileConfig(config.config_file_name)
-
-# add your model's MetaData object here
-# for 'autogenerate' support
-# from myapp import mymodel
-# target_metadata = mymodel.Base.metadata
-target_metadata = metadata
-
-# other values from the config, defined by the needs of env.py,
-# can be acquired:
-# my_important_option = config.get_main_option("my_important_option")
-# ... etc.
-
-
-def run_migrations_offline():
-    """Run migrations in 'offline' mode.
-
-    This configures the context with just a URL
-    and not an Engine, though an Engine is acceptable
-    here as well.  By skipping the Engine creation
-    we don't even need a DBAPI to be available.
-
-    Calls to context.execute() here emit the given string to the
-    script output.
-
-    """
-    raise Exception('Not implemented or configured!')
-
-    url = config.get_main_option("sqlalchemy.url")
-    context.configure(
-        url=url, target_metadata=target_metadata, literal_binds=True)
-
-    with context.begin_transaction():
-        context.run_migrations()
-
-
-def run_migrations_online():
-    """Run migrations in 'online' mode.
-
-    In this scenario we need to create an Engine
-    and associate a connection with the context.
-
-    """
-    ini_section = config.get_section(config.config_ini_section)
-    overrides = context.get_x_argument(as_dictionary=True)
-    for override in overrides:
-        ini_section[override] = overrides[override]
-
-    connectable = engine_from_config(
-        ini_section,
-        prefix='sqlalchemy.',
-        poolclass=pool.NullPool)
-
-    with connectable.connect() as connection:
-        context.configure(
-            connection=connection,
-            target_metadata=target_metadata,
-            compare_type=True,
-            compare_server_default=True,
-        )
-
-        with context.begin_transaction():
-            context.run_migrations()
-
-if context.is_offline_mode():
-    run_migrations_offline()
-else:
-    run_migrations_online()

core/data/alembic/script.py.mako

@@ -1,24 +0,0 @@
-"""${message}
-
-Revision ID: ${up_revision}
-Revises: ${down_revision | comma,n}
-Create Date: ${create_date}
-
-"""
-from alembic import op
-import sqlalchemy as sa
-${imports if imports else ""}
-
-# revision identifiers, used by Alembic.
-revision = ${repr(up_revision)}
-down_revision = ${repr(down_revision)}
-branch_labels = ${repr(branch_labels)}
-depends_on = ${repr(depends_on)}
-
-
-def upgrade():
-    ${upgrades if upgrades else "pass"}
-
-
-def downgrade():
-    ${downgrades if downgrades else "pass"}

core/data/alembic/versions/6a7e8277763b_gekichu_rating_tables.py

@@ -1,56 +0,0 @@
-"""GekiChu rating tables
-
-Revision ID: 6a7e8277763b
-Revises: d8950c7ce2fc
-Create Date: 2024-03-13 12:18:53.210018
-
-"""
-from alembic import op
-from sqlalchemy import Column, Integer, String
-
-
-# revision identifiers, used by Alembic.
-revision = '6a7e8277763b'
-down_revision = 'd8950c7ce2fc'
-branch_labels = None
-depends_on = None
-
-GEKICHU_RATING_TABLE_NAMES = [
-    "chuni_profile_rating",
-    "ongeki_profile_rating",
-]
-
-def upgrade():
-    for table_name in GEKICHU_RATING_TABLE_NAMES:
-        op.create_table(
-            table_name,
-            Column("id", Integer, primary_key=True, nullable=False),
-            Column("user", Integer, nullable=False),
-            Column("version", Integer, nullable=False),
-            Column("type", String(255), nullable=False),
-            Column("index", Integer, nullable=False),
-            Column("musicId", Integer),
-            Column("difficultId", Integer),
-            Column("romVersionCode", Integer),
-            Column("score", Integer),
-            mysql_charset="utf8mb4",
-        )
-        op.create_foreign_key(
-            None,
-            table_name,
-            "aime_user",
-            ["user"],
-            ["id"],
-            ondelete="cascade",
-            onupdate="cascade",
-        )
-        op.create_unique_constraint(
-            f"{table_name}_uk",
-            table_name,
-            ["user", "version", "type", "index"],
-        )
-
-
-def downgrade():
-    for table_name in GEKICHU_RATING_TABLE_NAMES:
-        op.drop_table(table_name)

core/data/alembic/versions/81e44dd6047a_mai2_buddies_support.py

@@ -1,68 +0,0 @@
-"""mai2_buddies_support
-
-Revision ID: 81e44dd6047a
-Revises: d8950c7ce2fc
-Create Date: 2024-03-12 19:10:37.063907
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-from sqlalchemy.dialects import mysql
-
-# revision identifiers, used by Alembic.
-revision = "81e44dd6047a"
-down_revision = "6a7e8277763b"
-branch_labels = None
-depends_on = None
-
-
-def upgrade():
-    op.create_table(
-        "mai2_playlog_2p",
-        sa.Column("id", sa.Integer(), nullable=False),
-        sa.Column("user", sa.Integer(), nullable=False),
-        sa.Column("userId1", sa.Integer(), nullable=True),
-        sa.Column("userId2", sa.Integer(), nullable=True),
-        sa.Column("userName1", sa.String(length=25), nullable=True),
-        sa.Column("userName2", sa.String(length=25), nullable=True),
-        sa.Column("regionId", sa.Integer(), nullable=True),
-        sa.Column("placeId", sa.Integer(), nullable=True),
-        sa.Column("user2pPlaylogDetailList", sa.JSON(), nullable=True),
-        sa.ForeignKeyConstraint(
-            ["user"], ["aime_user.id"], onupdate="cascade", ondelete="cascade"
-        ),
-        sa.PrimaryKeyConstraint("id"),
-        mysql_charset="utf8mb4",
-    )
-
-    op.add_column(
-        "mai2_playlog",
-        sa.Column(
-            "extBool1", sa.Boolean(), nullable=True, server_default=sa.text("NULL")
-        ),
-    )
-
-    op.add_column(
-        "mai2_profile_detail",
-        sa.Column(
-            "renameCredit", sa.Integer(), nullable=True, server_default=sa.text("NULL")
-        ),
-    )
-    op.add_column(
-        "mai2_profile_detail",
-        sa.Column(
-            "currentPlayCount",
-            sa.Integer(),
-            nullable=True,
-            server_default=sa.text("NULL"),
-        ),
-    )
-
-
-def downgrade():
-    op.drop_table("mai2_playlog_2p")
-
-    op.drop_column("mai2_playlog", "extBool1")
-    op.drop_column("mai2_profile_detail", "renameCredit")
-    op.drop_column("mai2_profile_detail", "currentPlayCount")

core/data/alembic/versions/835b862f9bf0_initial_migration.py

@@ -1,24 +0,0 @@
-"""Initial Migration
-
-Revision ID: 835b862f9bf0
-Revises: 
-Create Date: 2024-01-09 13:06:10.787432
-
-"""
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = '835b862f9bf0'
-down_revision = None
-branch_labels = None
-depends_on = None
-
-
-def upgrade():
-    pass
-
-
-def downgrade():
-    pass

core/data/alembic/versions/d8950c7ce2fc_remove_old_db_mgmt_system.py

@@ -1,29 +0,0 @@
-"""Remove old db mgmt system
-
-Revision ID: d8950c7ce2fc
-Revises: 835b862f9bf0
-Create Date: 2024-01-09 13:43:51.381175
-
-"""
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = 'd8950c7ce2fc'
-down_revision = '835b862f9bf0'
-branch_labels = None
-depends_on = None
-
-
-def upgrade():
-    op.drop_table("schema_versions")
-
-
-def downgrade():
-    op.create_table(
-        "schema_versions",
-        sa.Column("game", sa.String(4), primary_key=True, nullable=False),
-        sa.Column("version", sa.Integer, nullable=False, server_default="1"),
-        mysql_charset="utf8mb4",
-    )

core/data/alembic/versions/e4e8d89c9b02_idac_battle_gift_and_tips_added.py

@@ -1,83 +0,0 @@
-"""IDAC Battle Gift and Tips added
-
-Revision ID: e4e8d89c9b02
-Revises: 81e44dd6047a
-Create Date: 2024-04-01 17:49:50.009718
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-from sqlalchemy.dialects import mysql
-
-# revision identifiers, used by Alembic.
-revision = "e4e8d89c9b02"
-down_revision = "81e44dd6047a"
-branch_labels = None
-depends_on = None
-
-
-def upgrade():
-    op.create_table(
-        "idac_user_battle_gift",
-        sa.Column("id", sa.Integer(), nullable=False),
-        sa.Column("user", sa.Integer(), nullable=False),
-        sa.Column("battle_gift_event_id", sa.Integer(), nullable=True),
-        sa.Column("gift_id", sa.Integer(), nullable=True),
-        sa.Column("gift_status", sa.Integer(), nullable=True),
-        sa.Column(
-            "received_date",
-            sa.TIMESTAMP(),
-            server_default=sa.text("now()"),
-            nullable=True,
-        ),
-        sa.ForeignKeyConstraint(
-            ["user"], ["aime_user.id"], onupdate="cascade", ondelete="cascade"
-        ),
-        sa.PrimaryKeyConstraint("id"),
-        sa.UniqueConstraint(
-            "user", "battle_gift_event_id", "gift_id", name="idac_user_battle_gift_uk"
-        ),
-        mysql_charset="utf8mb4",
-    )
-
-    op.create_table(
-        "idac_profile_tips",
-        sa.Column("id", sa.Integer(), nullable=False),
-        sa.Column("user", sa.Integer(), nullable=False),
-        sa.Column("version", sa.Integer(), nullable=False),
-        sa.Column(
-            "tips_list",
-            sa.String(length=16),
-            server_default="QAAAAAAAAAAAAAAA",
-            nullable=True,
-        ),
-        sa.Column(
-            "timetrial_play_count", sa.Integer(), server_default="0", nullable=True
-        ),
-        sa.Column("story_play_count", sa.Integer(), server_default="0", nullable=True),
-        sa.Column(
-            "store_battle_play_count", sa.Integer(), server_default="0", nullable=True
-        ),
-        sa.Column(
-            "online_battle_play_count", sa.Integer(), server_default="0", nullable=True
-        ),
-        sa.Column(
-            "special_play_count", sa.Integer(), server_default="0", nullable=True
-        ),
-        sa.Column(
-            "challenge_play_count", sa.Integer(), server_default="0", nullable=True
-        ),
-        sa.Column("theory_play_count", sa.Integer(), server_default="0", nullable=True),
-        sa.ForeignKeyConstraint(
-            ["user"], ["aime_user.id"], onupdate="cascade", ondelete="cascade"
-        ),
-        sa.PrimaryKeyConstraint("id"),
-        sa.UniqueConstraint("user", "version", name="idac_profile_tips_uk"),
-        mysql_charset="utf8mb4",
-    )
-
-
-def downgrade():
-    op.drop_table("idac_user_battle_gift")
-    op.drop_table("idac_profile_tips")

core/data/cache.py

@@ -17,7 +17,7 @@ except ModuleNotFoundError:
 
 def cached(lifetime: int = 10, extra_key: Any = None) -> Callable:
     def _cached(func: Callable) -> Callable:
-        if has_mc and (cfg and cfg.database.enable_memcached):
+        if has_mc:
             hostname = "127.0.0.1"
             if cfg:
                 hostname = cfg.database.memcached_host

core/data/database.py

@@ -1,14 +1,13 @@
 import logging, coloredlogs
-from typing import Optional
+from typing import Optional, Dict, List
 from sqlalchemy.orm import scoped_session, sessionmaker
+from sqlalchemy.exc import SQLAlchemyError
 from sqlalchemy import create_engine
 from logging.handlers import TimedRotatingFileHandler
-import os
+import importlib, os
 import secrets, string
 import bcrypt
 from hashlib import sha256
-import alembic.config
-import glob
 
 from core.config import CoreConfig
 from core.data.schema import *
@@ -16,6 +15,7 @@ from core.utils import Utils
 
 
 class Data:
+    current_schema_version = 6
     engine = None
     session = None
     user = None
@@ -77,173 +77,281 @@ class Data:
             )
             self.logger.handler_set = True  # type: ignore
 
-    def __alembic_cmd(self, command: str, *args: str) -> None:
-        old_dir = os.path.abspath(os.path.curdir)
-        base_dir = os.path.join(os.path.abspath(os.path.curdir), 'core', 'data', 'alembic')
-        alembicArgs = [
-            "-c",
-            os.path.join(base_dir, "alembic.ini"),
-            "-x",
-            f"script_location={base_dir}",
-            "-x",
-            f"sqlalchemy.url={self.__url}",
-            command,
-        ]
-        alembicArgs.extend(args)
-        os.chdir(base_dir)
-        alembic.config.main(argv=alembicArgs)
-        os.chdir(old_dir)
-
     def create_database(self):
         self.logger.info("Creating databases...")
-        metadata.create_all(
+        try:
-            self.engine,
+            metadata.create_all(self.__engine.connect())
-            checkfirst=True,
+        except SQLAlchemyError as e:
-        )
+            self.logger.error(f"Failed to create databases! {e}")
+            return
 
-        for _, mod in Utils.get_all_titles().items():
+        games = Utils.get_all_titles()
-            if hasattr(mod, "database"):
+        for game_dir, game_mod in games.items():
-                mod.database(self.config)
+            try:
-                metadata.create_all(
+                if hasattr(game_mod, "database") and hasattr(
-                    self.engine,
+                    game_mod, "current_schema_version"
-                    checkfirst=True,
+                ):
+                    game_mod.database(self.config)
+                metadata.create_all(self.__engine.connect())
+
+                self.base.touch_schema_ver(
+                    game_mod.current_schema_version, game_mod.game_codes[0]
                 )
 
-        # Stamp the end revision as if alembic had created it, so it can take off after this.
+            except Exception as e:
-        self.__alembic_cmd(
+                self.logger.warning(
-            "stamp",
+                    f"Could not load database schema from {game_dir} - {e}"
-            "head",
+                )
+
+        self.logger.info(f"Setting base_schema_ver to {self.current_schema_version}")
+        self.base.set_schema_ver(self.current_schema_version)
+
+        self.logger.info(
+            f"Setting user auto_incrememnt to {self.config.database.user_table_autoincrement_start}"
+        )
+        self.user.reset_autoincrement(
+            self.config.database.user_table_autoincrement_start
         )
 
-    def schema_upgrade(self, ver: str = None):
+    def recreate_database(self):
-        self.__alembic_cmd(
+        self.logger.info("Dropping all databases...")
-            "upgrade",
+        self.base.execute("SET FOREIGN_KEY_CHECKS=0")
-            "head" if not ver else ver,
+        try:
-        )
+            metadata.drop_all(self.__engine.connect())
+        except SQLAlchemyError as e:
+            self.logger.error(f"Failed to drop databases! {e}")
+            return
 
-    def schema_downgrade(self, ver: str):
+        for root, dirs, files in os.walk("./titles"):
-        self.__alembic_cmd(
+            for dir in dirs:
-            "downgrade",
+                if not dir.startswith("__"):
-            ver,
+                    try:
-        )
+                        mod = importlib.import_module(f"titles.{dir}")
 
-    async def create_owner(self, email: Optional[str] = None, code: Optional[str] = "00000000000000000000") -> None:
+                        try:
+                            if hasattr(mod, "database"):
+                                mod.database(self.config)
+                            metadata.drop_all(self.__engine.connect())
+
+                        except Exception as e:
+                            self.logger.warning(
+                                f"Could not load database schema from {dir} - {e}"
+                            )
+
+                    except ImportError as e:
+                        self.logger.warning(
+                            f"Failed to load database schema dir {dir} - {e}"
+                        )
+            break
+
+        self.base.execute("SET FOREIGN_KEY_CHECKS=1")
+
+        self.create_database()
+
+    def migrate_database(self, game: str, version: Optional[int], action: str) -> None:
+        old_ver = self.base.get_schema_ver(game)
+        sql = ""
+        if version is None:
+            if not game == "CORE":
+                titles = Utils.get_all_titles()
+
+                for folder, mod in titles.items():
+                    if not mod.game_codes[0] == game:
+                        continue
+
+                    if hasattr(mod, "current_schema_version"):
+                        version = mod.current_schema_version
+
+                    else:
+                        self.logger.warning(
+                            f"current_schema_version not found for {folder}"
+                        )
+
+            else:
+                version = self.current_schema_version
+
+        if version is None:
+            self.logger.warning(
+                f"Could not determine latest version for {game}, please specify --version"
+            )
+
+        if old_ver is None:
+            self.logger.error(
+                f"Schema for game {game} does not exist, did you run the creation script?"
+            )
+            return
+
+        if old_ver == version:
+            self.logger.info(
+                f"Schema for game {game} is already version {old_ver}, nothing to do"
+            )
+            return
+
+        if action == "upgrade":
+            for x in range(old_ver, version):
+                if not os.path.exists(
+                    f"core/data/schema/versions/{game.upper()}_{x + 1}_{action}.sql"
+                ):
+                    self.logger.error(
+                        f"Could not find {action} script {game.upper()}_{x + 1}_{action}.sql in core/data/schema/versions folder"
+                    )
+                    return
+
+                with open(
+                    f"core/data/schema/versions/{game.upper()}_{x + 1}_{action}.sql",
+                    "r",
+                    encoding="utf-8",
+                ) as f:
+                    sql = f.read()
+
+                result = self.base.execute(sql)
+                if result is None:
+                    self.logger.error("Error execuing sql script!")
+                    return None
+
+        else:
+            for x in range(old_ver, version, -1):
+                if not os.path.exists(
+                    f"core/data/schema/versions/{game.upper()}_{x - 1}_{action}.sql"
+                ):
+                    self.logger.error(
+                        f"Could not find {action} script {game.upper()}_{x - 1}_{action}.sql in core/data/schema/versions folder"
+                    )
+                    return
+
+                with open(
+                    f"core/data/schema/versions/{game.upper()}_{x - 1}_{action}.sql",
+                    "r",
+                    encoding="utf-8",
+                ) as f:
+                    sql = f.read()
+
+                result = self.base.execute(sql)
+                if result is None:
+                    self.logger.error("Error execuing sql script!")
+                    return None
+
+        result = self.base.set_schema_ver(version, game)
+        if result is None:
+            self.logger.error("Error setting version in schema_version table!")
+            return None
+
+        self.logger.info(f"Successfully migrated {game} to schema version {version}")
+
+    def create_owner(self, email: Optional[str] = None) -> None:
         pw = "".join(
             secrets.choice(string.ascii_letters + string.digits) for i in range(20)
         )
         hash = bcrypt.hashpw(pw.encode(), bcrypt.gensalt())
 
-        user_id = await self.user.create_user(username="sysowner", email=email, password=hash.decode(), permission=255)
+        user_id = self.user.create_user(email=email, permission=255, password=hash)
         if user_id is None:
             self.logger.error(f"Failed to create owner with email {email}")
             return
 
-        card_id = await self.card.create_card(user_id, code)
+        card_id = self.card.create_card(user_id, "00000000000000000000")
         if card_id is None:
             self.logger.error(f"Failed to create card for owner with id {user_id}")
             return
 
         self.logger.warning(
-            f"Successfully created owner with email {email}, access code {code}, and password {pw} Make sure to change this password and assign a real card ASAP!"
+            f"Successfully created owner with email {email}, access code 00000000000000000000, and password {pw} Make sure to change this password and assign a real card ASAP!"
         )
 
-    async def migrate(self) -> None:
+    def migrate_card(self, old_ac: str, new_ac: str, should_force: bool) -> None:
-        exist = await self.base.execute("SELECT * FROM alembic_version")
+        if old_ac == new_ac:
-        if exist is not None:
+            self.logger.error("Both access codes are the same!")
-            self.logger.warn("No need to migrate as you have already migrated to alembic. If you are trying to upgrade the schema, use `upgrade` instead!")
             return
 
-        self.logger.info("Upgrading to latest with legacy system")
+        new_card = self.card.get_card_by_access_code(new_ac)
-        if not await self.legacy_upgrade():
+        if new_card is None:
-            self.logger.warn("No need to migrate as you have already deleted the old schema_versions system. If you are trying to upgrade the schema, use `upgrade` instead!")
+            self.card.update_access_code(old_ac, new_ac)
-            return
-        self.logger.info("Done")
-        
-        self.logger.info("Stamp with initial revision")
-        self.__alembic_cmd(
-            "stamp",
-            "835b862f9bf0",
-        )
-
-        self.logger.info("Upgrade")
-        self.__alembic_cmd(
-            "upgrade",
-            "head",
-        )
-    
-    async def legacy_upgrade(self) -> bool:
-        vers = await self.base.execute("SELECT * FROM schema_versions")
-        if vers is None:
-            self.logger.warn("Cannot legacy upgrade, schema_versions table unavailable!")
-            return False
-        
-        db_vers = {}
-        vers_list = vers.fetchall()
-        for x in vers_list:
-            db_vers[x['game']] = x['version']
-        
-        core_now_ver = int(db_vers['CORE']) + 1
-        while os.path.exists(f"core/data/schema/versions/CORE_{core_now_ver}_upgrade.sql"):
-            with open(f"core/data/schema/versions/CORE_{core_now_ver}_upgrade.sql", "r") as f:
-                result = await self.base.execute(f.read())
-                
-                if result is None:
-                    self.logger.error(f"Invalid upgrade script CORE_{core_now_ver}_upgrade.sql")
-                    break
-
-                result = await self.base.execute(f"UPDATE schema_versions SET version = {core_now_ver} WHERE game = 'CORE'")
-                if result is None:
-                    self.logger.error(f"Failed to update schema version for CORE to {core_now_ver}")
-                    break
-            
-            self.logger.info(f"Upgrade CORE to version {core_now_ver}")
-            core_now_ver += 1
-        
-        for _, mod in Utils.get_all_titles().items():
-            game_codes = getattr(mod, "game_codes", [])
-            for game in game_codes:
-                if game not in db_vers:
-                    self.logger.warn(f"{game} does not have an antry in schema_versions, skipping")
-                    continue
-
-                now_ver = int(db_vers[game]) + 1
-                while os.path.exists(f"core/data/schema/versions/{game}_{now_ver}_upgrade.sql"):
-                    with open(f"core/data/schema/versions/{game}_{now_ver}_upgrade.sql", "r") as f:
-                        result = await self.base.execute(f.read())
-                        
-                        if result is None:
-                            self.logger.error(f"Invalid upgrade script {game}_{now_ver}_upgrade.sql")
-                            break
-
-                        result = await self.base.execute(f"UPDATE schema_versions SET version = {now_ver} WHERE game = '{game}'")
-                        if result is None:
-                            self.logger.error(f"Failed to update schema version for {game} to {now_ver}")
-                            break
-
-                    self.logger.info(f"Upgrade {game} to version {now_ver}")
-                    now_ver += 1
-        
-        return True
-
-    async def create_revision(self, message: str) -> None:
-        if not message:
-            self.logger.info("Message is required for create-revision")
             return
 
-        self.__alembic_cmd(
+        if not should_force:
-            "revision",
+            self.logger.warning(
-            "-m",
+                f"Card already exists for access code {new_ac} (id {new_card['id']}). If you wish to continue, rerun with the '--force' flag."
-            message,
+                f" All exiting data on the target card {new_ac} will be perminently erased and replaced with data from card {old_ac}."
-        )
+            )
-
-    async def create_revision_auto(self, message: str) -> None:
-        if not message:
-            self.logger.info("Message is required for create-revision")
             return
 
-        self.__alembic_cmd(
+        self.logger.info(
-            "revision",
+            f"All exiting data on the target card {new_ac} will be perminently erased and replaced with data from card {old_ac}."
-            "--autogenerate",
-            "-m",
-            message,
         )
+        self.card.delete_card(new_card["id"])
+        self.card.update_access_code(old_ac, new_ac)
+
+        hanging_user = self.user.get_user(new_card["user"])
+        if hanging_user["password"] is None:
+            self.logger.info(f"Delete hanging user {hanging_user['id']}")
+            self.user.delete_user(hanging_user["id"])
+
+    def delete_hanging_users(self) -> None:
+        """
+        Finds and deletes users that have not registered for the webui that have no cards assocated with them.
+        """
+        unreg_users = self.user.get_unregistered_users()
+        if unreg_users is None:
+            self.logger.error("Error occoured finding unregistered users")
+
+        for user in unreg_users:
+            cards = self.card.get_user_cards(user["id"])
+            if cards is None:
+                self.logger.error(f"Error getting cards for user {user['id']}")
+                continue
+
+            if not cards:
+                self.logger.info(f"Delete hanging user {user['id']}")
+                self.user.delete_user(user["id"])
+
+    def autoupgrade(self) -> None:
+        all_game_versions = self.base.get_all_schema_vers()
+        if all_game_versions is None:
+            self.logger.warning("Failed to get schema versions")
+            return
+
+        all_games = Utils.get_all_titles()
+        all_games_list: Dict[str, int] = {}
+        for _, mod in all_games.items():
+            if hasattr(mod, "current_schema_version"):
+                all_games_list[mod.game_codes[0]] = mod.current_schema_version
+
+        for x in all_game_versions:
+            failed = False
+            game = x["game"].upper()
+            update_ver = int(x["version"])
+            latest_ver = all_games_list.get(game, 1)
+            if game == "CORE":
+                latest_ver = self.current_schema_version
+
+            if update_ver == latest_ver:
+                self.logger.info(f"{game} is already latest version")
+                continue
+
+            for y in range(update_ver + 1, latest_ver + 1):
+                if os.path.exists(f"core/data/schema/versions/{game}_{y}_upgrade.sql"):
+                    with open(
+                        f"core/data/schema/versions/{game}_{y}_upgrade.sql",
+                        "r",
+                        encoding="utf-8",
+                    ) as f:
+                        sql = f.read()
+
+                    result = self.base.execute(sql)
+                    if result is None:
+                        self.logger.error(
+                            f"Error execuing sql script for game {game} v{y}!"
+                        )
+                        failed = True
+                        break
+                else:
+                    self.logger.warning(f"Could not find script {game}_{y}_upgrade.sql")
+                    failed = True
+
+            if not failed:
+                self.base.set_schema_ver(latest_ver, game)
+    
+    def show_versions(self) -> None:
+        all_game_versions = self.base.get_all_schema_vers()
+        for ver in all_game_versions:
+            self.logger.info(f"{ver['game']} -> v{ver['version']}")

core/data/schema/arcade.py

@@ -41,6 +41,7 @@ machine = Table(
     Column("country", String(3)),  # overwrites if not null
     Column("timezone", String(255)),
     Column("ota_enable", Boolean),
+    Column("is_cab", Boolean),
     Column("memo", String(255)),
     Column("is_cab", Boolean),
     Column("data", JSON),
@@ -69,7 +70,7 @@ arcade_owner = Table(
 
 
 class ArcadeData(BaseData):
-    async def get_machine(self, serial: str = None, id: int = None) -> Optional[Row]:
+    def get_machine(self, serial: str = None, id: int = None) -> Optional[Row]:
         if serial is not None:
             serial = serial.replace("-", "")
             if len(serial) == 11:
@@ -89,12 +90,12 @@ class ArcadeData(BaseData):
             self.logger.error(f"{__name__ }: Need either serial or ID to look up!")
             return None
 
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return None
         return result.fetchone()
 
-    async def create_machine(
+    def put_machine(
         self,
         arcade_id: int,
         serial: str = "",
@@ -102,21 +103,21 @@ class ArcadeData(BaseData):
         game: str = None,
         is_cab: bool = False,
     ) -> Optional[int]:
-        if not arcade_id:
+        if arcade_id:
             self.logger.error(f"{__name__ }: Need arcade id!")
             return None
 
         sql = machine.insert().values(
-            arcade=arcade_id, serial=serial, board=board, game=game, is_cab=is_cab
+            arcade=arcade_id, keychip=serial, board=board, game=game, is_cab=is_cab
         )
 
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return None
         return result.lastrowid
 
-    async def set_machine_serial(self, machine_id: int, serial: str) -> None:
+    def set_machine_serial(self, machine_id: int, serial: str) -> None:
-        result = await self.execute(
+        result = self.execute(
             machine.update(machine.c.id == machine_id).values(keychip=serial)
         )
         if result is None:
@@ -125,8 +126,8 @@ class ArcadeData(BaseData):
             )
         return result.lastrowid
 
-    async def set_machine_boardid(self, machine_id: int, boardid: str) -> None:
+    def set_machine_boardid(self, machine_id: int, boardid: str) -> None:
-        result = await self.execute(
+        result = self.execute(
             machine.update(machine.c.id == machine_id).values(board=boardid)
         )
         if result is None:
@@ -134,29 +135,29 @@ class ArcadeData(BaseData):
                 f"Failed to update board id for machine {machine_id} -> {boardid}"
             )
 
-    async def get_arcade(self, id: int) -> Optional[Row]:
+    def get_arcade(self, id: int) -> Optional[Row]:
         sql = arcade.select(arcade.c.id == id)
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return None
         return result.fetchone()
     
-    async def get_arcade_machines(self, id: int) -> Optional[List[Row]]:
+    def get_arcade_machines(self, id: int) -> Optional[List[Row]]:
         sql = machine.select(machine.c.arcade == id)
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return None
         return result.fetchall()
 
-    async def create_arcade(
+    def put_arcade(
         self,
-        name: str = None,
+        name: str,
         nickname: str = None,
         country: str = "JPN",
         country_id: int = 1,
         state: str = "",
         city: str = "",
-        region_id: int = 1,
+        regional_id: int = 1,
     ) -> Optional[int]:
         if nickname is None:
             nickname = name
@@ -168,46 +169,46 @@ class ArcadeData(BaseData):
             country_id=country_id,
             state=state,
             city=city,
-            region_id=region_id,
+            regional_id=regional_id,
         )
 
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return None
         return result.lastrowid
 
-    async def get_arcades_managed_by_user(self, user_id: int) -> Optional[List[Row]]:
+    def get_arcades_managed_by_user(self, user_id: int) -> Optional[List[Row]]:
         sql = select(arcade).join(arcade_owner, arcade_owner.c.arcade == arcade.c.id).where(arcade_owner.c.user == user_id)
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return False
         return result.fetchall()
     
-    async def get_manager_permissions(self, user_id: int, arcade_id: int) -> Optional[int]:
+    def get_manager_permissions(self, user_id: int, arcade_id: int) -> Optional[int]:
         sql = select(arcade_owner.c.permissions).where(and_(arcade_owner.c.user == user_id, arcade_owner.c.arcade == arcade_id))
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return False
         return result.fetchone()
 
-    async def get_arcade_owners(self, arcade_id: int) -> Optional[Row]:
+    def get_arcade_owners(self, arcade_id: int) -> Optional[Row]:
         sql = select(arcade_owner).where(arcade_owner.c.arcade == arcade_id)
 
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return None
         return result.fetchall()
 
-    async def add_arcade_owner(self, arcade_id: int, user_id: int) -> None:
+    def add_arcade_owner(self, arcade_id: int, user_id: int) -> None:
         sql = insert(arcade_owner).values(arcade=arcade_id, user=user_id)
 
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return None
         return result.lastrowid
 
-    def format_serial( # TODO: Actual serial stuff
+    def format_serial(
-        self, platform_code: str, platform_rev: int, serial_num: int, append: int = 8888
+        self, platform_code: str, platform_rev: int, serial_num: int, append: int = 4152
     ) -> str:
         return f"{platform_code}{platform_rev:02d}A{serial_num:04d}{append:04d}"  # 0x41 = A, 0x52 = R
 
@@ -217,16 +218,9 @@ class ArcadeData(BaseData):
         
         return True
 
-    async def get_arcade_by_name(self, name: str) -> Optional[List[Row]]:
+    def find_arcade_by_name(self, name: str) -> List[Row]:
         sql = arcade.select(or_(arcade.c.name.like(f"%{name}%"), arcade.c.nickname.like(f"%{name}%")))
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
-            return None
+            return False
-        return result.fetchall()
-
-    async def get_arcades_by_ip(self, ip: str) -> Optional[List[Row]]:
-        sql = arcade.select().where(arcade.c.ip == ip)
-        result = await self.execute(sql)
-        if result is None:
-            return None
         return result.fetchall()

core/data/schema/base.py

@@ -15,6 +15,14 @@ from core.config import CoreConfig
 
 metadata = MetaData()
 
+schema_ver = Table(
+    "schema_versions",
+    metadata,
+    Column("game", String(4), primary_key=True, nullable=False),
+    Column("version", Integer, nullable=False, server_default="1"),
+    mysql_charset="utf8mb4",
+)
+
 event_log = Table(
     "event_log",
     metadata,
@@ -35,11 +43,11 @@ class BaseData:
         self.conn = conn
         self.logger = logging.getLogger("database")
 
-    async def execute(self, sql: str, opts: Dict[str, Any] = {}) -> Optional[CursorResult]:
+    def execute(self, sql: str, opts: Dict[str, Any] = {}) -> Optional[CursorResult]:
         res = None
 
         try:
-            self.logger.debug(f"SQL Execute: {''.join(str(sql).splitlines())}")
+            self.logger.info(f"SQL Execute: {''.join(str(sql).splitlines())}")
             res = self.conn.execute(text(sql), opts)
 
         except SQLAlchemyError as e:
@@ -74,7 +82,52 @@ class BaseData:
         """
         return randrange(10000, 9999999)
 
-    async def log_event(
+    def get_all_schema_vers(self) -> Optional[List[Row]]:
+        sql = select(schema_ver)
+
+        result = self.execute(sql)
+        if result is None:
+            return None
+        return result.fetchall()
+
+    def get_schema_ver(self, game: str) -> Optional[int]:
+        sql = select(schema_ver).where(schema_ver.c.game == game)
+
+        result = self.execute(sql)
+        if result is None:
+            return None
+
+        row = result.fetchone()
+        if row is None:
+            return None
+
+        return row["version"]
+    
+    def touch_schema_ver(self, ver: int, game: str = "CORE") -> Optional[int]:
+        sql = insert(schema_ver).values(game=game, version=ver)
+        conflict = sql.on_duplicate_key_update(version=schema_ver.c.version)
+
+        result = self.execute(conflict)
+        if result is None:
+            self.logger.error(
+                f"Failed to update schema version for game {game} (v{ver})"
+            )
+            return None
+        return result.lastrowid
+
+    def set_schema_ver(self, ver: int, game: str = "CORE") -> Optional[int]:
+        sql = insert(schema_ver).values(game=game, version=ver)
+        conflict = sql.on_duplicate_key_update(version=ver)
+
+        result = self.execute(conflict)
+        if result is None:
+            self.logger.error(
+                f"Failed to update schema version for game {game} (v{ver})"
+            )
+            return None
+        return result.lastrowid
+
+    def log_event(
         self, system: str, type: str, severity: int, message: str, details: Dict = {}
     ) -> Optional[int]:
         sql = event_log.insert().values(
@@ -84,7 +137,7 @@ class BaseData:
             message=message,
             details=json.dumps(details),
         )
-        result = await self.execute(sql)
+        result = self.execute(sql)
 
         if result is None:
             self.logger.error(
@@ -94,9 +147,9 @@ class BaseData:
 
         return result.lastrowid
 
-    async def get_event_log(self, entries: int = 100) -> Optional[List[Dict]]:
+    def get_event_log(self, entries: int = 100) -> Optional[List[Dict]]:
         sql = event_log.select().limit(entries).all()
-        result = await self.execute(sql)
+        result = self.execute(sql)
 
         if result is None:
             return None
@@ -104,8 +157,6 @@ class BaseData:
 
     def fix_bools(self, data: Dict) -> Dict:
         for k, v in data.items():
-            if k == "userName" or k == "teamName":
-                continue
             if type(v) == str and v.lower() == "true":
                 data[k] = True
             elif type(v) == str and v.lower() == "false":

core/data/schema/card.py

@@ -27,101 +27,70 @@ aime_card = Table(
 
 
 class CardData(BaseData):
-    async def get_card_by_access_code(self, access_code: str) -> Optional[Row]:
+    def get_card_by_access_code(self, access_code: str) -> Optional[Row]:
         sql = aime_card.select(aime_card.c.access_code == access_code)
 
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return None
         return result.fetchone()
 
-    async def get_card_by_id(self, card_id: int) -> Optional[Row]:
+    def get_card_by_id(self, card_id: int) -> Optional[Row]:
         sql = aime_card.select(aime_card.c.id == card_id)
 
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return None
         return result.fetchone()
 
-    async def update_access_code(self, old_ac: str, new_ac: str) -> None:
+    def update_access_code(self, old_ac: str, new_ac: str) -> None:
         sql = aime_card.update(aime_card.c.access_code == old_ac).values(
             access_code=new_ac
         )
 
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             self.logger.error(
                 f"Failed to change card access code from {old_ac} to {new_ac}"
             )
 
-    async def get_user_id_from_card(self, access_code: str) -> Optional[int]:
+    def get_user_id_from_card(self, access_code: str) -> Optional[int]:
         """
         Given a 20 digit access code as a string, get the user id associated with that card
         """
-        card = await self.get_card_by_access_code(access_code)
+        card = self.get_card_by_access_code(access_code)
         if card is None:
             return None
 
         return int(card["user"])
 
-    async def get_card_banned(self, access_code: str) -> Optional[bool]:
+    def delete_card(self, card_id: int) -> None:
-        """
-        Given a 20 digit access code as a string, check if the card is banned
-        """
-        card = await self.get_card_by_access_code(access_code)
-        if card is None:
-            return None
-        if card["is_banned"]:
-            return True
-        return False
-    
-    async def get_card_locked(self, access_code: str) -> Optional[bool]:
-        """
-        Given a 20 digit access code as a string, check if the card is locked
-        """
-        card = await self.get_card_by_access_code(access_code)
-        if card is None:
-            return None
-        if card["is_locked"]:
-            return True
-        return False
-
-    async def delete_card(self, card_id: int) -> None:
         sql = aime_card.delete(aime_card.c.id == card_id)
 
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             self.logger.error(f"Failed to delete card with id {card_id}")
 
-    async def get_user_cards(self, aime_id: int) -> Optional[List[Row]]:
+    def get_user_cards(self, aime_id: int) -> Optional[List[Row]]:
         """
         Returns all cards owned by a user
         """
         sql = aime_card.select(aime_card.c.user == aime_id)
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return None
         return result.fetchall()
 
-    async def create_card(self, user_id: int, access_code: str) -> Optional[int]:
+    def create_card(self, user_id: int, access_code: str) -> Optional[int]:
         """
         Given a aime_user id and a 20 digit access code as a string, create a card and return the ID if successful
         """
         sql = aime_card.insert().values(user=user_id, access_code=access_code)
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return None
         return result.lastrowid
 
-    async def update_card_last_login(self, access_code: str) -> None:
-        sql = aime_card.update(aime_card.c.access_code == access_code).values(
-            last_login_date=func.now()
-        )
-        
-        result = await self.execute(sql)
-        if result is None:
-            self.logger.warn(f"Failed to update last login time for {access_code}")
-    
     def to_access_code(self, luid: str) -> str:
         """
         Given a felica cards internal 16 hex character luid, convert it to a 0-padded 20 digit access code as a string

core/data/schema/user.py

@@ -1,3 +1,4 @@
+from enum import Enum
 from typing import Optional, List
 from sqlalchemy import Table, Column
 from sqlalchemy.types import Integer, String, TIMESTAMP
@@ -23,8 +24,15 @@ aime_user = Table(
     mysql_charset="utf8mb4",
 )
 
+
+class PermissionBits(Enum):
+    PermUser = 1
+    PermMod = 2
+    PermSysAdmin = 4
+
+
 class UserData(BaseData):
-    async def create_user(
+    def create_user(
         self,
         id: int = None,
         username: str = None,
@@ -52,20 +60,20 @@ class UserData(BaseData):
             username=username, email=email, password=password, permissions=permission
         )
 
-        result = await self.execute(conflict)
+        result = self.execute(conflict)
         if result is None:
             return None
         return result.lastrowid
 
-    async def get_user(self, user_id: int) -> Optional[Row]:
+    def get_user(self, user_id: int) -> Optional[Row]:
         sql = select(aime_user).where(aime_user.c.id == user_id)
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return False
         return result.fetchone()
 
-    async def check_password(self, user_id: int, passwd: bytes = None) -> bool:
+    def check_password(self, user_id: int, passwd: bytes = None) -> bool:
-        usr = await self.get_user(user_id)
+        usr = self.get_user(user_id)
         if usr is None:
             return False
 
@@ -77,46 +85,39 @@ class UserData(BaseData):
 
         return bcrypt.checkpw(passwd, usr["password"].encode())
 
-    async def delete_user(self, user_id: int) -> None:
+    def reset_autoincrement(self, ai_value: int) -> None:
+        # ALTER TABLE isn't in sqlalchemy so we do this the ugly way
+        sql = f"ALTER TABLE aime_user AUTO_INCREMENT={ai_value}"
+        self.execute(sql)
+
+    def delete_user(self, user_id: int) -> None:
         sql = aime_user.delete(aime_user.c.id == user_id)
 
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             self.logger.error(f"Failed to delete user with id {user_id}")
 
-    async def get_unregistered_users(self) -> List[Row]:
+    def get_unregistered_users(self) -> List[Row]:
         """
         Returns a list of users who have not registered with the webui. They may or may not have cards.
         """
         sql = select(aime_user).where(aime_user.c.password == None)
 
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return None
         return result.fetchall()
 
-    async def find_user_by_email(self, email: str) -> Row:
+    def find_user_by_email(self, email: str) -> Row:
         sql = select(aime_user).where(aime_user.c.email == email)
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return False
         return result.fetchone()
 
-    async def find_user_by_username(self, username: str) -> List[Row]:
+    def find_user_by_username(self, username: str) -> List[Row]:
         sql = aime_user.select(aime_user.c.username.like(f"%{username}%"))
-        result = await self.execute(sql)
+        result = self.execute(sql)
         if result is None:
             return False
         return result.fetchall()
-
-    async def change_password(self, user_id: int, new_passwd: str) -> bool:
-        sql = aime_user.update(aime_user.c.id == user_id).values(password = new_passwd)
-
-        result = await self.execute(sql)
-        return result is not None
-
-    async def change_username(self, user_id: int, new_name: str) -> bool:
-        sql = aime_user.update(aime_user.c.id == user_id).values(username = new_name)
-
-        result = await self.execute(sql)
-        return result is not None

core/data/schema/versions/SBZV_5_rollback.sql

@@ -1,2 +0,0 @@
-ALTER TABLE diva_profile
-    DROP skn_eqp;

core/data/schema/versions/SBZV_6_upgrade.sql

@@ -1,2 +0,0 @@
-ALTER TABLE diva_profile
-    ADD skn_eqp INT NOT NULL DEFAULT 0;

core/data/schema/versions/SDBT_4_rollback.sql

@@ -1,12 +0,0 @@
-SET FOREIGN_KEY_CHECKS = 0;
-
-ALTER TABLE chuni_score_playlog
-  CHANGE COLUMN isClear isClear TINYINT(1) NULL DEFAULT NULL;
-
-ALTER TABLE chuni_score_best
-  CHANGE COLUMN isSuccess isSuccess TINYINT(1) NULL DEFAULT NULL ;
-
-ALTER TABLE chuni_score_playlog
-  DROP COLUMN ticketId;
-
-SET FOREIGN_KEY_CHECKS = 1;

core/data/schema/versions/SDBT_5_upgrade.sql

@@ -1,12 +0,0 @@
-SET FOREIGN_KEY_CHECKS = 0;
-
-ALTER TABLE chuni_score_playlog
-  CHANGE COLUMN isClear isClear TINYINT(6) NULL DEFAULT NULL;
-
-ALTER TABLE chuni_score_best
-  CHANGE COLUMN isSuccess isSuccess INT(11) NULL DEFAULT NULL ;
-
-ALTER TABLE chuni_score_playlog
-  ADD COLUMN ticketId INT(11) NULL AFTER machineType;
-
-SET FOREIGN_KEY_CHECKS = 1;

core/data/schema/versions/SDDT_5_rollback.sql

@@ -1,22 +0,0 @@
-SET FOREIGN_KEY_CHECKS=0;
-
-ALTER TABLE ongeki_user_event_point DROP COLUMN version;
-ALTER TABLE ongeki_user_event_point DROP COLUMN `rank`;
-ALTER TABLE ongeki_user_event_point DROP COLUMN `type`;
-ALTER TABLE ongeki_user_event_point DROP COLUMN date;
-
-ALTER TABLE ongeki_user_tech_event DROP COLUMN version;
-
-ALTER TABLE ongeki_user_mission_point DROP COLUMN version;
-
-ALTER TABLE ongeki_static_events DROP COLUMN endDate;
-
-DROP TABLE ongeki_tech_event_ranking;
-DROP TABLE ongeki_static_music_ranking_list;
-DROP TABLE ongeki_static_rewards;
-DROP TABLE ongeki_static_present_list;
-DROP TABLE ongeki_static_tech_music;
-DROP TABLE ongeki_static_client_testmode;
-DROP TABLE ongeki_static_game_point;
-
-SET FOREIGN_KEY_CHECKS=1;

core/data/schema/versions/SDDT_6_upgrade.sql

@@ -1,98 +0,0 @@
-SET FOREIGN_KEY_CHECKS=0;
-
-ALTER TABLE ongeki_user_event_point ADD COLUMN version INTEGER NOT NULL;
-ALTER TABLE ongeki_user_event_point ADD COLUMN `rank` INTEGER;
-ALTER TABLE ongeki_user_event_point ADD COLUMN `type` INTEGER NOT NULL;
-ALTER TABLE ongeki_user_event_point ADD COLUMN date VARCHAR(25);
-
-ALTER TABLE ongeki_user_tech_event ADD COLUMN version INTEGER NOT NULL;
-
-ALTER TABLE ongeki_user_mission_point ADD COLUMN version INTEGER NOT NULL;
-
-ALTER TABLE ongeki_static_events ADD COLUMN endDate TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP;
-
-CREATE TABLE ongeki_tech_event_ranking (
-    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
-    user INT NOT NULL,
-    version INT NOT NULL,
-    date VARCHAR(25),
-    eventId INT NOT NULL,
-    `rank` INT,
-    totalPlatinumScore INT NOT NULL,
-    totalTechScore INT NOT NULL,
-    UNIQUE KEY ongeki_tech_event_ranking_uk (user, eventId),
-    CONSTRAINT ongeki_tech_event_ranking_ibfk1 FOREIGN KEY (user) REFERENCES aime_user(id) ON DELETE CASCADE ON UPDATE CASCADE
-);
-
-CREATE TABLE ongeki_static_music_ranking_list (
-    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
-    version INT NOT NULL,
-    musicId INT NOT NULL,
-    point INT NOT NULL,
-    userName VARCHAR(255),
-    UNIQUE KEY ongeki_static_music_ranking_list_uk (version, musicId)
-);
-
-CREATE TABLE ongeki_static_rewards (
-    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
-    version INT NOT NULL,
-    rewardId INT NOT NULL,
-    rewardName VARCHAR(255) NOT NULL,
-    itemKind INT NOT NULL,
-    itemId INT NOT NULL,
-    UNIQUE KEY ongeki_tech_event_ranking_uk (version, rewardId)
-);
-
-CREATE TABLE ongeki_static_present_list (
-    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
-    version INT NOT NULL,
-    presentId INT NOT NULL,
-    presentName VARCHAR(255) NOT NULL,
-    rewardId INT NOT NULL,
-    stock INT NOT NULL,
-    message VARCHAR(255),
-    startDate VARCHAR(25) NOT NULL,
-    endDate VARCHAR(25) NOT NULL,
-    UNIQUE KEY ongeki_static_present_list_uk (version, presentId, rewardId)
-);
-
-CREATE TABLE ongeki_static_tech_music (
-    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
-    version INT NOT NULL,
-    eventId INT NOT NULL,
-    musicId INT NOT NULL,
-    level INT NOT NULL,
-    UNIQUE KEY ongeki_static_tech_music_uk (version, musicId, eventId)
-);
-
-CREATE TABLE ongeki_static_client_testmode (
-    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
-    regionId INT NOT NULL,
-    placeId INT NOT NULL,
-    clientId VARCHAR(11) NOT NULL,
-    updateDate TIMESTAMP NOT NULL,
-    isDelivery BOOLEAN NOT NULL,
-    groupId INT NOT NULL,
-    groupRole INT NOT NULL,
-    continueMode INT NOT NULL,
-    selectMusicTime INT NOT NULL,
-    advertiseVolume INT NOT NULL,
-    eventMode INT NOT NULL,
-    eventMusicNum INT NOT NULL,
-    patternGp INT NOT NULL,
-    limitGp INT NOT NULL,
-    maxLeverMovable INT NOT NULL,
-    minLeverMovable INT NOT NULL,
-    UNIQUE KEY ongeki_static_client_testmode_uk (clientId)
-);
-
-CREATE TABLE ongeki_static_game_point (
-    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
-    `type` INT NOT NULL,
-    cost INT NOT NULL,
-    startDate VARCHAR(25) NOT NULL DEFAULT "2000-01-01 05:00:00.0",
-    endDate VARCHAR(25) NOT NULL DEFAULT "2099-01-01 05:00:00.0",
-    UNIQUE KEY ongeki_static_game_point_uk (`type`)
-);
-
-SET FOREIGN_KEY_CHECKS=1;

core/data/schema/versions/SDEZ_7_rollback.sql

@@ -1,10 +0,0 @@
-ALTER TABLE mai2_profile_detail
-    DROP COLUMN mapStock;
-
-ALTER TABLE mai2_profile_extend
-    DROP COLUMN selectResultScoreViewType;
-
-ALTER TABLE mai2_profile_option
-    DROP COLUMN outFrameType,
-    DROP COLUMN touchVolume,
-    DROP COLUMN breakSlideVolume;

core/data/schema/versions/SDEZ_8_upgrade.sql

@@ -1,10 +0,0 @@
-ALTER TABLE mai2_profile_detail
-    ADD mapStock INT NULL AFTER playCount;
-
-ALTER TABLE mai2_profile_extend
-    ADD selectResultScoreViewType INT NULL AFTER selectResultDetails;
-
-ALTER TABLE mai2_profile_option
-    ADD outFrameType INT NULL AFTER dispCenter,
-    ADD touchVolume INT NULL AFTER slideVolume,
-    ADD breakSlideVolume INT NULL AFTER slideVolume;

core/data/schema/versions/SDGT_1_rollback.sql

@@ -1,54 +0,0 @@
-SET FOREIGN_KEY_CHECKS=0;
-
--- WARNING: This script is NOT idempotent! MAKE A BACKUP BEFORE RUNNING THIS SCRIPT!
-
--- Drop UK idac_user_vs_info_uk
-ALTER TABLE idac_user_vs_info
-DROP FOREIGN KEY idac_user_vs_info_ibfk_1,
-DROP INDEX idac_user_vs_info_uk;
-
--- Drop the new columns added to the original table
-ALTER TABLE idac_user_vs_info
-DROP COLUMN battle_mode,
-DROP COLUMN invalid,
-DROP COLUMN str,
-DROP COLUMN str_now,
-DROP COLUMN lose_now;
-
--- Add back the old columns to the original table
-ALTER TABLE idac_user_vs_info
-ADD COLUMN group_key VARCHAR(25),
-ADD COLUMN win_flg INT,
-ADD COLUMN style_car_id INT,
-ADD COLUMN course_id INT,
-ADD COLUMN course_day INT,
-ADD COLUMN players_num INT,
-ADD COLUMN winning INT,
-ADD COLUMN advantage_1 INT,
-ADD COLUMN advantage_2 INT,
-ADD COLUMN advantage_3 INT,
-ADD COLUMN advantage_4 INT,
-ADD COLUMN select_course_id INT,
-ADD COLUMN select_course_day INT,
-ADD COLUMN select_course_random INT,
-ADD COLUMN matching_success_sec INT,
-ADD COLUMN boost_flag INT;
-
--- Delete the data from the original table where group_key is NULL
-DELETE FROM idac_user_vs_info
-WHERE group_key IS NULL;
-
--- Insert data back to the original table from idac_user_vs_course_info
-INSERT INTO idac_user_vs_info (user, group_key, win_flg, style_car_id, course_id, course_day, players_num, winning, advantage_1, advantage_2, advantage_3, advantage_4, select_course_id, select_course_day, select_course_random, matching_success_sec, boost_flag, vs_history, break_count, break_penalty_flag)
-SELECT user, CONCAT(FLOOR(RAND()*(99999999999999-10000000000000+1)+10000000000000), 'A69E01A8888'), 0, 0, course_id, 0, 0, vs_cnt, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
-FROM idac_user_vs_course_info;
-
--- Add back the constraints and indexes to the original table
-ALTER TABLE idac_user_vs_info
-ADD CONSTRAINT idac_user_vs_info_ibfk_1 FOREIGN KEY (user) REFERENCES aime_user(id) ON DELETE CASCADE ON UPDATE CASCADE,
-ADD UNIQUE KEY idac_user_vs_info_uk (user, group_key);
-
--- Drop the new table idac_user_vs_course_info
-DROP TABLE IF EXISTS idac_user_vs_course_info;
-
-SET FOREIGN_KEY_CHECKS=1;

core/data/schema/versions/SDGT_2_upgrade.sql

@@ -1,71 +0,0 @@
-SET FOREIGN_KEY_CHECKS=0;
-
--- WARNING: This script is NOT idempotent! MAKE A BACKUP BEFORE RUNNING THIS SCRIPT!
-
--- Create the new table idac_user_vs_course_info
-CREATE TABLE idac_user_vs_course_info (
-    id INT PRIMARY KEY AUTO_INCREMENT,
-    user INT,
-    battle_mode INT,
-    course_id INT,
-    vs_cnt INT,
-    vs_win INT,
-    CONSTRAINT idac_user_vs_course_info_fk FOREIGN KEY (user) REFERENCES aime_user(id) ON DELETE CASCADE ON UPDATE CASCADE,
-    UNIQUE KEY idac_user_vs_course_info_uk (user, battle_mode, course_id)
-) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
-
--- Insert data from the original table to the new tables
-INSERT INTO idac_user_vs_course_info (user, battle_mode, course_id, vs_cnt, vs_win)
-SELECT user, 1 as battle_mode, course_id, COUNT(winning) as vs_cnt, SUM(win_flg) as vs_win
-FROM idac_user_vs_info
-GROUP BY user, course_id;
-
--- Drop UK idac_user_vs_info_uk
-ALTER TABLE idac_user_vs_info
-DROP FOREIGN KEY idac_user_vs_info_ibfk_1,
-DROP INDEX idac_user_vs_info_uk;
-
--- Drop/Add the old columns from the original table
-ALTER TABLE idac_user_vs_info
-DROP COLUMN group_key,
-DROP COLUMN win_flg,
-DROP COLUMN style_car_id,
-DROP COLUMN course_id,
-DROP COLUMN course_day,
-DROP COLUMN players_num,
-DROP COLUMN winning,
-DROP COLUMN advantage_1,
-DROP COLUMN advantage_2,
-DROP COLUMN advantage_3,
-DROP COLUMN advantage_4,
-DROP COLUMN select_course_id,
-DROP COLUMN select_course_day,
-DROP COLUMN select_course_random,
-DROP COLUMN matching_success_sec,
-DROP COLUMN boost_flag,
-
-ADD COLUMN battle_mode TINYINT UNSIGNED DEFAULT 1 NOT NULL AFTER user,
-ADD COLUMN invalid INT DEFAULT 0,
-ADD COLUMN str INT DEFAULT 0,
-ADD COLUMN str_now INT DEFAULT 0,
-ADD COLUMN lose_now INT DEFAULT 0;
-
--- Create a temporary table to store the records you want to keep
-CREATE TEMPORARY TABLE temp_table AS
-SELECT MIN(id) AS min_id
-FROM idac_user_vs_info
-GROUP BY battle_mode, user;
-
--- Delete records from the original table based on the temporary table
-DELETE FROM idac_user_vs_info
-WHERE id NOT IN (SELECT min_id FROM temp_table);
-
--- Drop the temporary table
-DROP TEMPORARY TABLE IF EXISTS temp_table;
-
--- Add UK idac_user_vs_info_uk
-ALTER TABLE idac_user_vs_info
-ADD CONSTRAINT idac_user_vs_info_ibfk_1 FOREIGN KEY (user) REFERENCES aime_user(id) ON DELETE CASCADE ON UPDATE CASCADE,
-ADD UNIQUE KEY idac_user_vs_info_uk (user, battle_mode);
-
-SET FOREIGN_KEY_CHECKS=1;

core/frontend/arcade/index.jinja

@@ -0,0 +1,4 @@
+{% extends "core/frontend/index.jinja" %}
+{% block content %}
+<h1>{{ arcade.name }}</h1>
+{% endblock content %}

core/frontend/gate/create.jinja

@@ -0,0 +1,44 @@
+{% extends "core/frontend/index.jinja" %}
+{% block content %}
+<div class="row justify-content-md-center form-signin">
+    <div class="col col-lg-4">
+        <form id="create" action="/gate/gate.create" method="post">
+            <h1 class="h3 mb-3 fw-normal">Sign-up</h1>
+
+            <div class="form-floating mb-3">
+                <input class="form-control" name="access-code" id="access-code" type="text"
+                    placeholder="00000000000000000000" value={{ code }} maxlength="20" required>
+                <label for="access-code">Access Code*</label>
+            </div>
+            <div class="form-floating mb-3">
+                <input type="text" class="form-control" name="username" id="username" placeholder="username">
+                <label for="username">Username*</label>
+            </div>
+            <div class="form-floating mb-3">
+                <input type="email" class="form-control" name="email" id="email" placeholder="example@example.com">
+                <label for="email">E-Mail*</label>
+            </div>
+            <div class="form-floating mb-3">
+                <input type="password" class="form-control" name="passwd" id="passwd" placeholder="Password">
+                <label for="passwd">Password*</label>
+            </div>
+            
+            <!--
+            <div class="form-check text-start my-3">
+                <input class="form-check-input" type="checkbox" value="remember-me" id="remember-me">
+                <label class="form-check-label" for="remember-me">
+                    Remember me
+                </label>
+            </div>
+            -->
+
+            <div class="alert alert-warning" role="alert">
+                If you have not registered a card with this server, you cannot create a WebUI account!
+            </div>
+
+            <button class="btn btn-primary w-100 py-2" type="submit">Sign-up</button>
+            <!-- <p class="mt-5 mb-3 text-body-secondary">© 2023 ARTEMiS</p>-->
+        </form>
+    </div>
+</div>
+{% endblock content %}

core/frontend/gate/gate.jinja

@@ -0,0 +1,52 @@
+{% extends "core/frontend/index.jinja" %}
+{% block content %}
+<style>
+    .form-signin input[type="text"] {
+        margin-bottom: -1px;
+        border-bottom-right-radius: 0;
+        border-bottom-left-radius: 0;
+    }
+
+    .form-signin input[type="password"] {
+        margin-bottom: 10px;
+        border-top-left-radius: 0;
+        border-top-right-radius: 0;
+    }
+</style>
+<div class="row justify-content-md-center form-signin">
+    <div class="col col-lg-4">
+        <form id="login" action="/gate/gate.login" method="post">
+            <h1 class="h3 mb-3 fw-normal">Login</h1>
+
+            {% include "core/frontend/widgets/err_banner.jinja" %}
+
+            <div class="form-floating">
+                <input form="login" class="form-control" name="access-code" id="access-code" type="text"
+                    placeholder="00000000000000000000" maxlength="20" required>
+                <label for="access-code">Access Code</label>
+            </div>
+            <div class="form-floating">
+                <input form="login" type="password" class="form-control" name="passwd" id="passwd" placeholder="Password">
+                <label for="passwd">Password</label>
+            </div>
+            
+            <!--
+            <div class="form-check text-start my-3">
+                <input class="form-check-input" type="checkbox" value="remember-me" id="remember-me">
+                <label class="form-check-label" for="remember-me">
+                    Remember me
+                </label>
+            </div>
+            -->
+
+            <div class="alert alert-info" role="alert">
+                To register for the WebUI, type in the <strong>access code</strong> of your card, as shown in a game, and leave the
+                password field blank.
+            </div>
+
+            <button class="btn btn-primary w-100 py-2" type="submit">Sign in</button>
+            <!-- <p class="mt-5 mb-3 text-body-secondary">© 2023 ARTEMiS</p>-->
+        </form>
+    </div>
+</div>
+{% endblock content %}

core/frontend/index.jinja

@@ -0,0 +1,25 @@
+<!DOCTYPE html>
+<html lang="en" data-bs-theme="dark">
+
+<head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+    <title>{{ title }}</title>
+    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.1/dist/css/bootstrap.min.css" rel="stylesheet"
+        integrity="sha384-4bw+/aepP/YC94hEpVNVgiZdgIC5+VKNBQNGCHeKRQN+PtmoHDEXuppvnDJzQIu9" crossorigin="anonymous">
+    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.10.5/font/bootstrap-icons.css">
+</head>
+
+<body>
+    {% include "core/frontend/widgets/topbar.jinja" %}
+    <div class="container">
+        {% block content %}
+        <h1>{{ server_name }}</h1>
+        {% endblock content %}
+    </div>
+    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.1/dist/js/bootstrap.bundle.min.js"
+        integrity="sha384-HwwvtgBNo3bZJJLYd8oVXjrBZt8cqVSpeBNS5n7C8IVInixGAoxmnlMuBnhbgrkm"
+        crossorigin="anonymous"></script>
+</body>
+
+</html>

core/frontend/machine/index.jinja

@@ -0,0 +1,5 @@
+{% extends "core/frontend/index.jinja" %}
+{% block content %}
+{% include "core/frontend/widgets/err_banner.jinja" %}
+<h1>Machine Management</h1>
+{% endblock content %}

core/frontend/sys/index.jinja

@@ -0,0 +1,98 @@
+{% extends "core/frontend/index.jinja" %}
+{% block content %}
+<h1>System Management</h1>
+
+<div class="row" id="rowForm">
+    {% if sesh.permissions >= 2 %}
+    <div class="col-sm-6" style="max-width: 25%;">
+        <form id="usrLookup" name="usrLookup" action="/sys/lookup.user" class="form-inline">
+            <h3>User Search</h3>
+            <div class="form-group">
+                <label for="usrEmail">Email address</label>
+                <input type="email" class="form-control" id="usrEmail" name="usrEmail" aria-describedby="emailHelp">
+            </div>
+            OR
+            <div class="form-group">
+                <label for="usrName">Username</label>
+                <input type="text" class="form-control" id="usrName" name="usrName">
+            </div>
+            OR
+            <div class="form-group">
+                <label for="usrId">User ID</label>
+                <input type="number" class="form-control" id="usrId" name="usrId">
+            </div>
+            <br />
+            <button type="submit" class="btn btn-primary">Search</button>
+        </form>
+    </div>
+    {% endif %}
+    {% if sesh.permissions >= 4 %}
+    <div class="col-sm-6" style="max-width: 25%;">
+        <form id="arcadeLookup" name="arcadeLookup" action="/sys/lookup.arcade" class="form-inline" >
+            <h3>Arcade Search</h3>
+            <div class="form-group">
+                <label for="arcadeName">Arcade Name</label>
+                <input type="text" class="form-control" id="arcadeName" name="arcadeName">
+            </div>
+            OR
+            <div class="form-group">
+                <label for="arcadeId">Arcade ID</label>
+                <input type="number" class="form-control" id="arcadeId" name="arcadeId">
+            </div>
+            OR
+            <div class="form-group">
+                <label for="arcadeUser">Owner User ID</label>
+                <input type="number" class="form-control" id="arcadeUser" name="arcadeUser">
+            </div>
+            <br />
+            <button type="submit" class="btn btn-primary">Search</button>
+        </form>
+    </div>
+    <div class="col-sm-6" style="max-width: 25%;">
+        <form id="cabLookup" name="cabLookup" action="/sys/lookup.cab" class="form-inline" >
+            <h3>Machine Search</h3>
+            <div class="form-group">
+                <label for="cabSerial">Machine Serial</label>
+                <input type="text" class="form-control" id="cabSerial" name="cabSerial">
+            </div>
+            OR
+            <div class="form-group">
+                <label for="cabId">Machine ID</label>
+                <input type="number" class="form-control" id="cabId" name="cabId">
+            </div>
+            OR
+            <div class="form-group">
+                <label for="cabAcId">Arcade ID</label>
+                <input type="number" class="form-control" id="cabAcId" name="cabAcId">
+            </div>
+            <br />
+            <button type="submit" class="btn btn-primary">Search</button>
+        </form>
+    </div>
+    {% endif %}
+</div>
+<div class="row" id="rowResult" style="margin: 10px;">
+    {% if sesh.permissions >= 2 %}
+    <div id="userSearchResult" class="col-sm-6" style="max-width: 25%;">
+        {% for usr in usrlist %}
+        <pre><a href=/user/{{ usr.id }}>{{ usr.id }} | {{ usr.username }}</a></pre>
+        {% endfor %}
+    </div>
+    {% endif %}
+    {% if sesh.permissions >= 4 %}
+    <div id="arcadeSearchResult" class="col-sm-6" style="max-width: 25%;">
+        {% for ac in aclist %}
+        <pre><a href=/arcade/{{ ac.id }}>{{ ac.id }} | {{ ac.name }}</a></pre>
+        {% endfor %}
+    </div
+    ><div id="cabSearchResult" class="col-sm-6" style="max-width: 25%;">
+        {% for cab in cablist %}
+        <a href=/cab/{{ cab.id }}><pre>{{ cab.id }} | {{ cab.game if cab.game is defined else "ANY "  }} | {{ cab.serial }}</pre></a>
+        {% endfor %}
+    </div>
+    {% endif %}
+</div>
+<div class="row" id="rowAdd">
+
+</div>
+{% endblock content %}

core/frontend/user/index.jinja

@@ -0,0 +1,126 @@
+{% extends "core/frontend/index.jinja" %}
+{% block content %}
+<h1>{{ sesh["username"] }}'s Account</h1>
+
+<div class="card mb-3">
+    <div class="card-body">
+        <h3 class="card-title">Cards</h3>
+        <!--<h4 class="card-subtitle mb-2 text-body-secondary">Card subtitle</h4>-->
+        <p class="card-text">All aime cards are listed here for the given user.</p>
+        <a href="#" data-bs-toggle="modal" data-bs-target="#card-add" class="btn btn-primary mb-3">Add Card</a>
+        {% if cards is defined and cards|length > 0 %}
+        <div class="container">
+            <div class="row">
+                <div class="col-12">
+                    <table class="table table-hover">
+                        <thead>
+                            <tr>
+                                <th scope="col">#</th>
+                                <th scope="col">Access Code</th>
+                                <th scope="col">Status</th>
+                                <th scope="col">Actions</th>
+                            </tr>
+                        </thead>
+                        <tbody>
+                            {% for c in cards %}
+                            <tr class="align-middle">
+                                <th scope="row">{{ c.index }}</th>
+                                <td>{{ c.access_code }}</td>
+                                <td>
+                                    {% if c.status == 'Active'%}
+                                    <span class="badge rounded-pill text-bg-success">Active</span>
+                                    {% elif c.status == 'Locked' %}
+                                    <span class="badge rounded-pill text-bg-warning">Locked</span>
+                                    {% endif %}
+                                </td>
+                                <td>
+                                    {% if c.status == 'Active'%}
+                                    <button type="button" class="btn btn-warning btn-sm">Lock</i></button>
+                                    {% elif c.status == 'Locked' %}
+                                    <button type="button" class="btn btn-success btn-sm">Unlock</i></button>
+                                    {% endif %}
+                                    <button type="button" class="btn btn-danger btn-sm">Delete</i></button>
+                                </td>
+                            </tr>
+                            {% endfor %}
+                        </tbody>
+                    </table>
+                </div>
+            </div>
+        </div>
+        {% else %}
+        <div class="alert alert-warning" role="alert">
+            You have no cards registered to your account. Please add one.
+        </div>
+        {% endif %}
+        <!--<a href="#" data-bs-toggle="modal" data-bs-target="#card-add" class="card-link">Add Card</a>-->
+    </div>
+</div>
+
+<div class="card">
+    <div class="card-body">
+        <h3 class="card-title">Arcades</h3>
+        <!--<h4 class="card-subtitle mb-2 text-body-secondary">Card subtitle</h4>-->
+        <p class="card-text">All arcades connected to the given account are listed here.</p>
+        {% if arcades is defined and arcades|length > 0 %}
+        <div class="container">
+            <div class="row">
+                <div class="col-12">
+                    <table class="table table-hover">
+                        <thead>
+                            <tr>
+                                <th scope="col">#</th>
+                                <th scope="col">Name</th>
+                            </tr>
+                        </thead>
+                        <tbody>
+                            {% for a in arcades %}
+                            <tr class="align-middle clickable-row" data-href=/arcade/{{ a.id }}>
+                                <th scope="row">{{ a.index }}</th>
+                                <td>{{ a.name }}</td>
+                            </tr>
+                            {% endfor %}
+                        </tbody>
+                    </table>
+                </div>
+            </div>
+        </div>
+        {% else %}
+        <div class="alert alert-info" role="alert">
+            You have no arcades connected to your account.
+        </div>
+        {% endif %}
+    </div>
+</div>
+
+<div class="modal fade" id="card-add" tabindex="-1" aria-labelledby="card-add-label" aria-hidden="true">
+    <div class="modal-dialog">
+        <div class="modal-content">
+            <div class="modal-header">
+                <h1 class="modal-title fs-5" id="card-add-label">Add Card</h1>
+                <button type="button" class="btn-close" data-bs-dismiss="modal" aria-label="Close"></button>
+            </div>
+            <div class="modal-body">
+                <h3 class="fs-5">How to:</h3>
+                Scan your card on any networked game and press the "View Access Code" button (varies by game) and enter
+                the 20
+                digit code below.
+                <hr>
+                FOR AMUSE IC CARDS: DO NOT ENTER THE CODE SHOWN ON THE BACK OF THE CARD ITSELF OR IT WILL NOT WORK!
+                <form>
+                    <div class="form-floating mt-3">
+                        <label for="access-code" class="col-form-label">Access Code</label>
+                        <input maxlength="20" type="text" required pattern="[0-9]{20}" class="form-control"
+                            id="access-code">
+                    </div>
+                </form>
+            </div>
+            <div class="modal-footer">
+                <button type="button" class="btn btn-secondary" data-bs-dismiss="modal">Close</button>
+                <button type="button" class="btn btn-primary">Add</button>
+            </div>
+        </div>
+    </div>
+</div>
+
+{% endblock content %}

core/frontend/widgets/err_banner.jinja

@@ -0,0 +1,17 @@
+{% if error > 0 %}
+<div class="alert alert-danger" role="alert">
+{% if error == 1 %}
+Card not registered, or wrong password
+{% elif error == 2 %}
+Missing or malformed access code
+{% elif error == 3 %}
+Failed to create user
+{% elif error == 4 %}
+Arcade not found
+{% elif error == 5 %}
+Machine not found
+{% else %}
+An unknown error occoured
+{% endif %}
+</div>
+{% endif %}

core/frontend/widgets/topbar.jinja

@@ -0,0 +1,56 @@
+<nav class="p-3 mb-3 navbar navbar-expand-lg bg-body-tertiary" aria-label="Thirteenth navbar example">
+    <div class="container">
+        <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarsExample11"
+            aria-controls="navbarsExample11" aria-expanded="false" aria-label="Toggle navigation">
+            <span class="navbar-toggler-icon"></span>
+        </button>
+
+        <div class="collapse navbar-collapse d-lg-flex" id="navbarsExample11">
+            <a class="navbar-brand col-lg-3 me-0" href= />ARTEMiS</a>
+            <ul class="navbar-nav col-lg-6 justify-content-lg-center nav-pills">
+                <li class="nav-item">
+                    <a class="nav-link {% if active_page == '/' %}active{% endif %}" aria-current="page" href= />Home</a>
+                </li>
+                {% for game in game_list %}
+                <li class="nav-item">
+                    <a class="nav-link {% if active_page == game.url %}active{% endif %}" href=/game/{{ game.url }}>{{ game.name }}</a>
+                </li>
+                {% endfor %}
+                <!--
+                    <li class="nav-item dropdown">
+                        <a class="nav-link dropdown-toggle" href="#" data-bs-toggle="dropdown"
+                            aria-expanded="false">Dropdown</a>
+                        <ul class="dropdown-menu">
+                            <li><a class="dropdown-item" href="#">Action</a></li>
+                            <li><a class="dropdown-item" href="#">Another action</a></li>
+                            <li><a class="dropdown-item" href="#">Something else here</a></li>
+                        </ul>
+                    </li>
+                    -->
+            </ul>
+            <div class="d-lg-flex col-lg-3 justify-content-lg-end">
+                {% if sesh is defined and sesh["user_id"] > 0 %}
+                <div class="btn-group dropdown">
+                    <a href="#" class="d-block link-body-emphasis text-decoration-none dropdown-toggle"
+                        data-bs-toggle="dropdown" aria-expanded="false">
+                       {{ sesh["username"] }}
+                    </a>
+                    <ul class="dropdown-menu text-small">
+                        {% if sesh["permissions"] >= 2 %}
+                        <li><a class="dropdown-item {% if active_page == 'sys' %}active{% endif %}" href="/sys">System</a></li>
+                        {% endif %}
+                        <li><a class="dropdown-item" href="#">Settings</a></li>
+                        <li><a class="dropdown-item {% if active_page == 'user' %}active{% endif %}" href="/user">Account</a></li>
+                        <li>
+                            <hr class="dropdown-divider">
+                        </li>
+                        <li><a class="dropdown-item" href="#">Sign out</a></li>
+                    </ul>
+                </div>
+                {% else %}
+                <a href="/gate"><button class="btn {% if active_page == 'gate' %}btn-primary{% else %}btn-outline-primary{% endif %}">Login</button></a>
+                {% endif %}
+            </div>
+        </div>
+    </div>
+</nav>

core/mucha.py

@@ -1,23 +1,21 @@
-from typing import Dict, Any, Optional
+from typing import Dict, Any, Optional, List
 import logging, coloredlogs
 from logging.handlers import TimedRotatingFileHandler
-from starlette.requests import Request
+from twisted.web import resource
-from starlette.responses import PlainTextResponse
+from twisted.web.http import Request
 from datetime import datetime
 from Crypto.Cipher import Blowfish
 import pytz
 
-from .config import CoreConfig
+from core import CoreConfig
-from .utils import Utils
+from core.utils import Utils
-from .title import TitleServlet
+
-from .data import Data
-from .const import *
 
 class MuchaServlet:
-    mucha_registry: Dict[str, Dict[str, str]] = {}
     def __init__(self, cfg: CoreConfig, cfg_dir: str) -> None:
         self.config = cfg
         self.config_dir = cfg_dir
+        self.mucha_registry: List[str] = []
 
         self.logger = logging.getLogger("mucha")
         log_fmt_str = "[%(asctime)s] Mucha | %(levelname)s | %(message)s"
@@ -39,147 +37,91 @@ class MuchaServlet:
         self.logger.setLevel(cfg.mucha.loglevel)
         coloredlogs.install(level=cfg.mucha.loglevel, logger=self.logger, fmt=log_fmt_str)
 
-        self.data = Data(cfg)
+        all_titles = Utils.get_all_titles()
 
-        for _, mod in TitleServlet.title_registry.items():
+        for _, mod in all_titles.items():
-            enabled, game_cds, netids = mod.get_mucha_info(self.config, self.config_dir)
+            if hasattr(mod, "index") and hasattr(mod.index, "get_mucha_info"):
-            if enabled:
+                enabled, game_cd = mod.index.get_mucha_info(
-                for x in range(len(game_cds)):
+                    self.config, self.config_dir
-                    self.mucha_registry[game_cds[x]] = { "netid_prefix": netids[x] }
+                )
+                if enabled:
+                    self.mucha_registry.append(game_cd)
 
         self.logger.info(f"Serving {len(self.mucha_registry)} games")
 
-    async def handle_boardauth(self, request: Request) -> bytes:
+    def handle_boardauth(self, request: Request, _: Dict) -> bytes:
-        bod = await request.body()
+        req_dict = self.mucha_preprocess(request.content.getvalue())
-        req_dict = self.mucha_preprocess(bod)
         client_ip = Utils.get_ip_addr(request)
 
         if req_dict is None:
             self.logger.error(
-                f"Error processing mucha request {bod}"
+                f"Error processing mucha request {request.content.getvalue()}"
             )
-            return PlainTextResponse("RESULTS=000")
+            return b"RESULTS=000"
 
         req = MuchaAuthRequest(req_dict)
-        self.logger.debug(f"Mucha request {vars(req)}")
+        self.logger.info(f"Boardauth request from {client_ip} for {req.gameVer}")        
-        
-        if not req.gameCd or not req.gameVer or not req.sendDate or not req.countryCd or not req.serialNum:
-            self.logger.warn(f"Missing required fields - {vars(req)}")
-            return PlainTextResponse("RESULTS=000")
-
-        minfo = self.mucha_registry.get(req.gameCd, {})
-
-        if not minfo:
-            self.logger.warning(f"Unknown gameCd {req.gameCd} from {client_ip}")
-            return PlainTextResponse("RESULTS=000")
-
-        b_key = b""
-        for x in range(8):
-            b_key += req.sendDate[(x - 1) & 7].encode()
-        
-        b_iv = b_key # what the fuck namco
-
-        cipher = Blowfish.new(b_key, Blowfish.MODE_CBC, b_iv)
-        try:
-            sn_decrypt = cipher.decrypt(bytes.fromhex(req.serialNum))[:12].decode()
-        except Exception as e:
-            self.logger.error(f"Decrypt SN {req.serialNum} failed! - {e}")
-            return PlainTextResponse("RESULTS=000")
-
-        self.logger.info(f"Boardauth request from {sn_decrypt} ({client_ip}) for {req.gameVer}")
-
-        resp = MuchaAuthResponse(
-            f"{self.config.server.hostname}{':' + str(self.config.server.port) if not self.config.server.is_using_proxy else ''}"
-        )
-
-        netid = minfo.get('netid_prefix', "ABxN") + sn_decrypt[5:]
-
-        cab = await self.data.arcade.get_machine(netid)
-        if cab:
-            arcade = await self.data.arcade.get_arcade(cab['id'])
-            if not arcade:
-                self.logger.error(f"Failed to get arcade with id {cab['id']}")
-                return PlainTextResponse("RESULTS=000")
-
-            resp.AREA_0 = arcade["region_id"] or AllnetJapanRegionId.AICHI.name
-            resp.AREA_0_EN = arcade["region_id"] or AllnetJapanRegionId.AICHI.name
-            resp.AREA_FULL_0 = arcade["region_id"] or AllnetJapanRegionId.AICHI.name
-            resp.AREA_FULL_0_EN = arcade["region_id"] or AllnetJapanRegionId.AICHI.name
-            
-            resp.AREA_1 = arcade["country"] or cab['country'] or AllnetCountryCode.JAPAN.value
-            resp.AREA_1_EN = arcade["country"] or cab['country'] or AllnetCountryCode.JAPAN.value
-            resp.AREA_FULL_1 = arcade["country"] or cab['country'] or AllnetCountryCode.JAPAN.value
-            resp.AREA_FULL_1_EN = arcade["country"] or cab['country'] or AllnetCountryCode.JAPAN.value
-
-            resp.AREA_2 = arcade["city"] if arcade["city"] else ""
-            resp.AREA_2_EN = arcade["city"] if arcade["city"] else ""
-            resp.AREA_FULL_2 = arcade["city"] if arcade["city"] else ""
-            resp.AREA_FULL_2_EN = arcade["city"] if arcade["city"] else ""
-
-            resp.AREA_3 = ""
-            resp.AREA_3_EN = ""
-            resp.AREA_FULL_3 = ""
-            resp.AREA_FULL_3_EN = ""
-
-            resp.PREFECTURE_ID = arcade['region_id']
-            resp.COUNTRY_CD = arcade['country'] or cab['country'] or AllnetCountryCode.JAPAN.value
-            resp.PLACE_ID = req.placeId if req.placeId else f"{arcade['country'] or cab['country'] or AllnetCountryCode.JAPAN.value}{arcade['id']:04X}"
-            resp.SHOP_NAME = arcade['name']
-            resp.SHOP_NAME_EN = arcade['name']
-            resp.SHOP_NICKNAME = arcade['nickname']
-            resp.SHOP_NICKNAME_EN = arcade['nickname']
-
-        elif self.config.server.allow_unregistered_serials:
-            self.logger.info(f"Allow unknown serial {netid} ({sn_decrypt}) to auth")
-        
-        else:
-            self.logger.warn(f'Auth failed for NetID {netid}')
-            return PlainTextResponse("RESULTS=000")
-
-        self.logger.debug(f"Mucha response {vars(resp)}")
-
-        return PlainTextResponse(self.mucha_postprocess(vars(resp)))
-
-    async def handle_updatecheck(self, request: Request) -> bytes:
-        bod = await request.body()
-        req_dict = self.mucha_preprocess(bod)
-        client_ip = Utils.get_ip_addr(request)
-
-        if req_dict is None:
-            self.logger.error(
-                f"Error processing mucha request {bod}"
-            )
-            return PlainTextResponse("RESULTS=000")
-
-        req = MuchaUpdateRequest(req_dict)
-        self.logger.info(f"Updatecheck request from {req.serialNum} ({client_ip}) for {req.gameVer}")        
         self.logger.debug(f"Mucha request {vars(req)}")
 
         if req.gameCd not in self.mucha_registry:
             self.logger.warning(f"Unknown gameCd {req.gameCd}")
-            return PlainTextResponse("RESULTS=000")
+            return b"RESULTS=000"
 
-        resp = MuchaUpdateResponse(req.gameVer, f"{self.config.server.hostname}{':' + str(self.config.server.port) if not self.config.server.is_using_proxy else ''}")
+        # TODO: Decrypt S/N
+        b_key = b""
+        for x in range(8):
+            b_key += req.sendDate[(x - 1) & 7].encode()
+
+        cipher = Blowfish.new(b_key, Blowfish.MODE_ECB)
+        sn_decrypt = cipher.decrypt(bytes.fromhex(req.serialNum))
+        self.logger.debug(f"Decrypt SN to {sn_decrypt.hex()}")
+
+        resp = MuchaAuthResponse(
+            f"{self.config.mucha.hostname}{':' + str(self.config.allnet.port) if self.config.server.is_develop else ''}"
+        )
 
         self.logger.debug(f"Mucha response {vars(resp)}")
 
-        return PlainTextResponse(self.mucha_postprocess(vars(resp)))
+        return self.mucha_postprocess(vars(resp))
 
-    async def handle_dlstate(self, request: Request) -> bytes:
+    def handle_updatecheck(self, request: Request, _: Dict) -> bytes:
-        bod = await request.body()
+        req_dict = self.mucha_preprocess(request.content.getvalue())
-        req_dict = self.mucha_preprocess(bod)
         client_ip = Utils.get_ip_addr(request)
 
         if req_dict is None:
             self.logger.error(
-                f"Error processing mucha request {bod}"
+                f"Error processing mucha request {request.content.getvalue()}"
             )
-            return PlainTextResponse("RESULTS=000")
+            return b"RESULTS=000"
+
+        req = MuchaUpdateRequest(req_dict)
+        self.logger.info(f"Updatecheck request from {client_ip} for {req.gameVer}")        
+        self.logger.debug(f"Mucha request {vars(req)}")
+
+        if req.gameCd not in self.mucha_registry:
+            self.logger.warning(f"Unknown gameCd {req.gameCd}")
+            return b"RESULTS=000"
+
+        resp = MuchaUpdateResponse(req.gameVer, f"{self.config.mucha.hostname}{':' + str(self.config.allnet.port) if self.config.server.is_develop else ''}")
+
+        self.logger.debug(f"Mucha response {vars(resp)}")
+
+        return self.mucha_postprocess(vars(resp))
+
+    def handle_dlstate(self, request: Request, _: Dict) -> bytes:
+        req_dict = self.mucha_preprocess(request.content.getvalue())
+        client_ip = Utils.get_ip_addr(request)
+
+        if req_dict is None:
+            self.logger.error(
+                f"Error processing mucha request {request.content.getvalue()}"
+            )
+            return b""
         
         req = MuchaDownloadStateRequest(req_dict)
-        self.logger.info(f"DownloadState request from {req.serialNum} ({client_ip}) for {req.gameCd} -> {req.updateVer}")        
+        self.logger.info(f"DownloadState request from {client_ip} for {req.gameCd} -> {req.updateVer}")        
         self.logger.debug(f"request {vars(req)}")
-        return PlainTextResponse("RESULTS=001")
+        return b"RESULTS=001"
 
     def mucha_preprocess(self, data: bytes) -> Optional[Dict]:
         try:
@@ -229,7 +171,7 @@ class MuchaAuthResponse:
         self.RESULTS = "001"
         self.AUTH_INTERVAL = "86400"
         self.SERVER_TIME = datetime.strftime(datetime.now(), "%Y%m%d%H%M")
-        self.SERVER_TIME_UTC = datetime.strftime(datetime.now(pytz.UTC), "%Y%m%d%H%M")
+        self.UTC_SERVER_TIME = datetime.strftime(datetime.now(pytz.UTC), "%Y%m%d%H%M")
 
         self.CHARGE_URL = f"https://{mucha_url}/charge/"
         self.FILE_URL = f"https://{mucha_url}/file/"

core/templates/arcade/index.jinja

@@ -1,19 +0,0 @@
-{% extends "core/templates/index.jinja" %}
-{% block content %}
-{% if arcade is defined %}
-<h1>{{ arcade.name }}</h1>
-<h2>PCBs assigned to this arcade <button class="btn btn-success" id="btn_add_cab" onclick="toggle_add_cab_form()">Add</button></h2>
-{% if success is defined and success == 3 %}
-<div style="background-color: #00AA00; padding: 20px; margin-bottom: 10px; width: 15%;">
-Cab added successfully
-</div>
-{% endif %}
-<ul style="font-size: 20px;">
-{% for c in arcade.cabs %}
-<li><a href="/cab/{{ c.id }}">{{ c.serial }}</a> ({{ c.game if c.game else "Any" }})&nbsp;<button class="btn btn-secondary" onclick="prep_edit_form()">Edit</button>&nbsp;<button class="btn-danger btn">Delete</button></li>
-{% endfor %}
-</ul>
-{% else %}
-<h3>Arcade Not Found</h3>
-{% endif %}
-{% endblock content %}

core/templates/gate/create.jinja

@@ -1,24 +0,0 @@
-{% extends "core/templates/index.jinja" %}
-{% block content %}
-<h1>Create User</h1>
-<form id="create" style="max-width: 240px; min-width: 10%;" action="/gate/gate.create" method="post">
-    <div class="form-group row">
-        <label for="access_code">Card Access Code</label><br>
-        <input class="form-control" name="access_code" id="access_code" type="text" placeholder="00000000000000000000" value={{ code }} maxlength="20" readonly>
-    </div>
-    <div class="form-group row">
-        <label for="username">Username</label><br>
-        <input id="username" class="form-control" name="username" type="text" placeholder="username">
-    </div>
-    <div class="form-group row">
-        <label for="email">Email</label><br>
-        <input id="email" class="form-control" name="email" type="email" placeholder="example@example.com">
-    </div>
-    <div class="form-group row">
-        <label for="passwd">Password</label><br>
-        <input id="passwd" class="form-control" name="passwd" type="password" placeholder="password">
-    </div>
-    <p></p>
-    <input id="submit" class="btn btn-primary" style="display: block; margin: 0 auto;" type="submit" value="Create">
-</form>
-{% endblock content %}