Merge pull request 'development' (#1 ) from development into main

Reviewed-on: #1
add link and logging
2026-04-23 06:28:06 +00:00 · 2026-04-23 08:25:52 +02:00 · 2026-04-22 20:39:59 +02:00 · 2026-04-20 12:43:40 +02:00 · 2026-04-20 12:42:54 +02:00
7 changed files with 446 additions and 135 deletions
@@ -675,3 +675,7 @@ fabric.properties
 .env
 lvenv
 wvenv 
 ## html doc
 *.html
@@ -7,6 +7,7 @@
    - loose 
    - passthrough
 - [x] profiles in a seperate .py file
 - Ok you see, I am too lazy to write the stuff which I completed here, so i just italizized it.
 ## To implement:
@@ -27,15 +28,15 @@ NOTE: That things which are in italic text are already implemented!
 ---
 ### Expland strict mode:
- [ ] Allow these:
+- *[x] Allow these:*
-    - html, head, body
+    - *html, head, body*
-    - title
+    - *title*
-    - meta (with restrictions)
+    - *meta (with restrictions)*
-    - link (for css only)
+    - *link (for css only)*
-    - table, tr, td, th
+    - *table, tr, td, th*
-    - any of crucuial html tags
+    - *any of crucuial html tags*
- [ ] restrict metas (allow):
+- [x] *restrict metas (allow):*
-    - allow only charset, name, content
+    - *allow only charset, name, content*
 - [ ] restrict metas (dont allow):
    - base
    - meta http-equiv
@@ -80,3 +81,144 @@ Lets maybe use bs4 for this, its fitting cause it can decompose HTML. Fixed!
 ### Around 30 minutes later:
 This issue is no longer relevant, it is fixed! using bs4.
 ## Development brach additions (overview of functionallity):
 As a example of the functionallity we use a command to call our sanizizer using the strict mode and we specifiy a configuration living in examples/config_example.json:
 ```bash
 cat ~/Documents/INF6B/html/wm/startseite_wm.html | python src/main.py --mode strict > out.html
 ```
 We output to a file called out.html, this is what the stdout of our programm looks like. if we grad stderr we get any debugging information aswell as errors / warnings / info
 ### command output:
 ```bash
 (venv) sebastian@kubuntu:~/Documents/sanigi-html$ cat ~/Documents/INF6B/html/wm/startseite_wm.html | python src/main.py --mode strict > out.html
 [ DEBUG@2026-04-20 12:32:50 ]: called sanitizer! using this as args: 
 <!DOCTYPE html>
 <html lang="en">
 <h1>Die Fußball-Weltmeisterschaft 2026</h1>
 <p id="h1_under">vom 11. Juni bis 19. Juli 2026 in Kanada, Mexiko und den USA</p>
 <img height="480" src="images/wmlogo.jpg" width="640"/>
 <ul>
 <li>Eröffnungsspiel:
            <ul>
 <li>Am 11. Juni 2026 um 21:00 Uhr im Aztekenstadion in Mexiko-Stadt (Mexiko vs. Südafrika)</li>
 </ul>
 </li>
 <li>Finale:
            <ul>
 <li>Am 19. Juli 2026 im MetLife Stadium in East Rutherford, New Jersey</li>
 </ul>
 </li>
 <li>Teilnehmer:
            <ul>
 <li>Erste WM mit 48 teilnehmenden Nationen</li>
 </ul>
 </li>
 <li>Format:
            <ul>
 <li>Insgesamt 104 Spiele in 12 Gruppen</li>
 <li>Austragung in 16 Stadien in drei Ländern</li>
 </ul>
 </li>
 <li>Zeitzonen:
            <ul>
 <li>Das Turnier erstreckt sich über vier verschiedene Zeitzonen in Nordamerika</li>
 </ul>
 </li>
 <li>Maskottchen:
            <ul>
 <li>Maple, Zayu &amp; Clutch</li>
 </ul>
 </li>
 </ul>
 </html>['html', 'body', 'header', 'main', 'footer', 'section', 'article', 'nav', 'aside', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'p', 'pre', 'code', 'blockquote', 'ul', 'ol', 'li', 'dl', 'dt', 'dd', 'br', 'hr', 'a', 'img', 'figure', 'figcaption', 'table', 'thead', 'tbody', 'tfoot', 'tr', 'th', 'td', 'strong', 'em', 'b', 'i', 'u', 'small', 'details', 'summary', 'time']
 ['http', 'https', 'mailto']
 True
 ```
 This is what the programm outputted to the stdout file (inserted into out.html):
 ```html
    <!DOCTYPE html>
    <html>
    <!-- sanigi-html parsed @ 2026-04-20 12:42:08.625481-->
    <head></head>
    <body>
 <h1>Die Fußball-Weltmeisterschaft 2026</h1>
 <p id="h1_under">vom 11. Juni bis 19. Juli 2026 in Kanada, Mexiko und den USA</p>
 <img height="480" width="640">
 <ul>
 <li>Eröffnungsspiel:
            <ul>
 <li>Am 11. Juni 2026 um 21:00 Uhr im Aztekenstadion in Mexiko-Stadt (Mexiko vs. Südafrika)</li>
 </ul>
 </li>
 <li>Finale:
            <ul>
 <li>Am 19. Juli 2026 im MetLife Stadium in East Rutherford, New Jersey</li>
 </ul>
 </li>
 <li>Teilnehmer:
            <ul>
 <li>Erste WM mit 48 teilnehmenden Nationen</li>
 </ul>
 </li>
 <li>Format:
            <ul>
 <li>Insgesamt 104 Spiele in 12 Gruppen</li>
 <li>Austragung in 16 Stadien in drei Ländern</li>
 </ul>
 </li>
 <li>Zeitzonen:
            <ul>
 <li>Das Turnier erstreckt sich über vier verschiedene Zeitzonen in Nordamerika</li>
 </ul>
 </li>
 <li>Maskottchen:
            <ul>
 <li>Maple, Zayu &amp; Clutch</li>
 </ul>
 </li>
 </ul>
 </body>
    </html>
 ```
 ### Fix (22/04/26):
 rewrite of sanizizer.py
 now output looks like this:
 ```html
 <!DOCTYPE html>
 <html>
 <!-- sanigi-html parsed @ 2026-04-22 20:23:01.043256 -->
 <head>
 <meta charset="utf-8">
 <meta content="width=device-width, initial-scale=1.0" name="viewport">
 <title>Document</title>
 </head>
 <body>
 <h1>Test</h1>
 <p>This is a test</p>
 </body>
 </html>
 ```
@@ -0,0 +1,63 @@
 #
 #   configuration.py
 #   AGPL-3.0    -   sanigi-html
 #   json config loader
 #
 from pathlib import Path
 import json
 def load_json_cfg(path: Path):
    if path == None:
        return {}
    if not path.exists():
        raise FileNotFoundError(path)
    with path.open("r", encoding="utf-8") as f:
        raw = json.load(f)
    def to_bool(variable, default=False):
        """
        basically just like a match case, but functionalized
        so we can call it more efficiently.
        """
        if isinstance(variable, bool): return variable
        if isinstance(variable, str): 
            return variable.lower() in ("1", "true", "yes", "no")
        return default
    def to_list(variable):
        if variable is None:
            return []
        if isinstance(variable, str):
            return [
                # very cheeky logic here!
                # this is a oneliner to split at ,
                s.strip() for s in variable.split(",") if s.strip()
            ]
        if isinstance(variable, list):
            return [
                # this is another example of cheeky logic!
                # its fun to figure out yourself, trust me
                str(x).strip() for x in variable if str(x).strip()
            ]
        return []
    configuration = {
        "allow_relative": to_bool(raw.get("ALLOW_RELATIVE_URLS"), True),
        "allowed_domains_default": to_list(raw.get("ALLOWED_DOMAINS")),
        "allowed_domains_img": to_list(raw.get("ALLOWED_DOMAINS_IMG")),
        "allowed_domains_link": to_list(raw.get("ALLOWED_DOMAINS_LINK")),
        "STRICT_overrides": raw.get("STRICT"),
        "LOOSE_overrides": raw.get("LOOSE"),
    }
    if not configuration["allowed_domains_img"]:
        configuration["allowed_domains_img"] = configuration["allowed_domains_default"]
    if not configuration["allowed_domains_link"]:
        configuration["allowed_domains_link"] = configuration["allowed_domains_default"]
    return configuration
@@ -0,0 +1,18 @@
 {
  "ALLOW_RELATIVE_URLS": true,
  "ALLOWED_DOMAINS": "example.com, static.example.com",
  "ALLOWED_DOMAINS_IMG": "images.example.com",
  "ALLOWED_DOMAINS_LINK": "example.com, partner.example.org",
  "STRICT": {
    "tags": ["p","b","i","ul","ol","li","br","h1","h2"],
    "protocols": [],
    "strip": true
  },
  "LOOSE": {
    "tags": ["p","b","i","u","em","strong","ul","ol","li","br","hr","span","div","a","img","h1","h2","h3"],
    "protocols": ["http","https"],
    "strip": true
  }
 }
@@ -7,22 +7,41 @@
 import sys
 import argparse
 from pathlib import Path
 from configuration import load_json_cfg 
 from profiles import build_profiles_from_cfg
 from sanitizer import sanitize
 def entry():
    parser = argparse.ArgumentParser(description="saningi-html sanitizer")
    parser.add_argument(
        "--mode", 
        choices=["strict","loose","passthrough"], 
        default="strict",
-        help="Strictness of Tags or HTML"
+        help="strictness"
    )
    parser.add_argument(
        "--config",
        type=Path,
        help="path to config json"
    )
    args = parser.parse_args()
    cfg = load_json_cfg(args.config)
    profile_dict = build_profiles_from_cfg(cfg)
    # cause i dont hate myself
    PROFILES = {
        "strict": profile_dict["STRICT"],
        "loose": profile_dict["LOOSE"],
    }
    # read from stdin and putout to stdout
    raw_html = sys.stdin.read()
-    clean_html = sanitize(raw_html, args.mode)
+    clean_html = sanitize(raw_html, args.mode, PROFILES)
    print(clean_html)
@@ -2,22 +2,15 @@
 #   src/profiles.py
 #   AGPL-3    -   saningi-html
 #
-#   presets for sanitization of HTML
+#   presets for sanitization of HTML;
 #   and newly the building of configurations
 #
 import os
 from dotenv import load_dotenv 
 from urllib.parse import urlparse
 from typing import Dict, Any, Optional
-# loadenv init
+def is_safe_url(url: str, allowed_domain=None, allow_relative: bool = True) -> bool:
 load_dotenv()
 def is_safe_url(url : str, allowed_domain=None, allow_relative : bool = True) -> bool:
    """
    Determine wether or not a URL is safe or not.
    Safe URL's are only local so /this/url
    Unsafe URL's are external ones like https://pornhub.com
    """
    if not url:
        return False
@@ -52,62 +45,105 @@ def create_url_filter(allowed_attrs, allowed_domains=None, allow_relative=True):
        return value
    return filter_func
-def get_domains(variable):
+def build_profiles_from_cfg(cfg: Optional[Dict[str, Any]] = None) -> Dict[str, Dict]:
    """
    This method esentially builds any configuration from a json config
    """
    def get_domains_from_env(variable):
        value = os.getenv(variable, "")
-    return [
+        return [d.strip() for d in value.split(",") if d.strip()]
        # return a list of domains, and split by comma
        d.strip() for d in value.split(",") if d.strip()
    ]
-# Setup any .env variables for python to use
+    if cfg is None:
        allow_relative = os.getenv("ALLOW_RELATIVE_URLS", "true").lower() == "true"
        allowed_domains_default = get_domains_from_env("ALLOWED_DOMAINS")
        allowed_domains_img = get_domains_from_env("ALLOWED_DOMAINS_IMG") or allowed_domains_default
        allowed_domains_link = get_domains_from_env("ALLOWED_DOMAINS_LINK") or allowed_domains_default
        strict_overrides = None
        loose_overrides = None
    else:
        allow_relative = cfg.get("allow_relative", True)
        allowed_domains_default = cfg.get("allowed_domains_default", [])
        allowed_domains_img = cfg.get("allowed_domains_img", allowed_domains_default)
        allowed_domains_link = cfg.get("allowed_domains_link", allowed_domains_default)
        strict_overrides = cfg.get("STRICT_overrides")
        loose_overrides = cfg.get("LOOSE_overrides")
-allow_relative = os.getenv("ALLOW_RELATIVE_URLS", "true").lower() == "true"
+    # like daddy :3
-
+    STRICT = {
 allowed_domains_default = get_domains("ALLOWED_DOMAINS")
 allowed_domains_img = get_domains("ALLOWED_DOMAINS_IMG") or allowed_domains_default
 allowed_domains_link = get_domains("ALLOWED_DOMAINS_LINK") or allowed_domains_default
 STRICT = {
    "tags" : [
        "p", "b", "i", "u", "em", "strong",
        "ul", "ol", "li",
        "br", "hr",
        "h1", "h2", "h3", "h4", "h5", "h6"
    ],
    "attributes": {
        "a": create_url_filter(
            ["href"],
            allowed_domains=allowed_domains_link,
            allow_relative=allow_relative
        ),
        "img": create_url_filter(
            ["src", "alt"],
            allowed_domains=allowed_domains_img,
            allow_relative=allow_relative
        ),
        "link": create_url_filter(
            ["href"],
            allowed_domains=allowed_domains_default,
            allow_relative=allow_relative
        ),
    },
    "protocols": [],
    "strip": True,
 }
 # like my ass :3
 LOOSE = {
        "tags": [
-        "p", "b", "i", "u", "em", "strong",
+            "html", "body", "header", "head", "title", "meta", "link", "body", "main", "footer", "section", "article", "nav",
-        "ul", "ol", "li",
+            "aside", "h1", "h2", "h3", "h4", "h5", "h6",
-        "br", "hr", "span", "div",
+            "p", "pre", "code", "blockquote",
-        "a", "img",
+            "ul", "ol", "li", "dl", "dt", "dd",
-        "h1", "h2", "h3", "h4", "h5", "h6"
+            "br", "hr",
            "a", "img", "figure", "figcaption",
            "table", "thead", "tbody", "tfoot", "tr", "th", "td",
            "strong", "em", "b", "i", "u", "small",
            "details", "summary", "time"
        ],
        "attributes": {
-        "a": ["href", "title"],
+            "a": create_url_filter(["href", "title", "rel", "target"], allowed_domains=allowed_domains_link, allow_relative=allow_relative),
-        "img": {"src","alt","style","loading"}
+            "img": create_url_filter(["src", "alt", "width", "height", "loading"], allowed_domains=allowed_domains_img, allow_relative=allow_relative),
            "figure": ["class"],
            "figcaption": ["class"],
            "table": ["summary", "class"],
            "th": ["scope", "colspan", "rowspan"],
            "td": ["colspan", "rowspan"],
            "link": ["href", "rel"],
            "time": ["datetime"],
            "meta": ["charset", "name", "content"],
            "html": ["lang"],
            "*": ["class", "id"] # limit
        },
-    "protocols": ["http", "https"],
+        "protocols": ["http", "https", "mailto"],
        "strip": True,
-}
+    }
    # like my ass :3
    LOOSE = {
        "tags": [
            "html", "body", "header", "head", "title", "meta", "link", "body", "main", "footer", "section", "article", "nav",
            "aside", "h1", "h2", "h3", "h4", "h5", "h6",
            "p", "pre", "code", "blockquote", "q",
            "ul", "ol", "li", "dl", "dt", "dd",
            "br", "hr", "span", "div",
            "a", "img", "figure", "figcaption",
            "table", "thead", "tbody", "tfoot", "tr", "th", "td",
            "strong", "em", "b", "i", "u", "small",
            "details", "summary", "time",
            "video", "audio", "source", "picture", "source",
            "caption"
        ],
        "attributes": {
            "a": ["href", "title", "rel", "target"],
            "img": ["src", "alt", "width", "height", "loading", "decoding", "style"],
            "figure": ["class", "style"],
            "figcaption": ["class", "style"],
            "table": ["summary", "class", "style"],
            "th": ["scope", "colspan", "rowspan", "style"],
            "td": ["colspan", "rowspan", "style"],
            "time": ["datetime"],
            "video": ["src", "controls", "width", "height", "poster"],
            "audio": ["src", "controls"],
            "source": ["src", "type"],
            "div": ["class", "id", "style"],
            "span": ["class", "id", "style"],
            "pre": ["class"],
            "code": ["class"],
            "link" : ["href", "rel"],
            "meta": ["charset", "name", "content"],
            "html": ["lang"],
            "*": ["class", "id", "style"]
        },
        "protocols": ["http", "https", "mailto", "data"],
        "strip": True,
    }
    # overrides
    if isinstance(strict_overrides, dict):
        STRICT.update(strict_overrides)
    if isinstance(loose_overrides, dict):
        LOOSE.update(loose_overrides)
    return {"STRICT": STRICT, "LOOSE": LOOSE}
@@ -3,20 +3,15 @@
 #   AGPL-3.0 - saningi-html
 #
 import bleach
-from bs4 import BeautifulSoup, Tag
+import datetime
-from profiles import STRICT, LOOSE
+from bs4 import BeautifulSoup
-from log.Logger import *
+from log.Logger import Logger
 logger = Logger()
 PROFILES = {
    "strict": STRICT,
    "loose": LOOSE,
 }
-
+def sanitize(html: str, mode: str, profiles: dict) -> str:
 def sanitize(html : str, mode : str) -> str:
    """
    Arguments:
        html : str  -> input html
@@ -26,50 +21,84 @@ def sanitize(html : str, mode : str) -> str:
        This function takes in HTML sanitzies it with bleach and returns clean HTML
    """
    # this should be used with caution! it is for trusted
    # individuals only
    if mode == "passthrough":
        logger.log_warning("You are about to not care about sanitizing ANY HTML. You are in passthrough mode.")
-        return html
+        return _wrap_document(html)
-    config = PROFILES.get(mode)
+    config = profiles.get(mode)
    if not config:
-        raise ValueError(f"Unknown mode! : {mode}")
+        raise ValueError(f"Unknown mode: {mode}")
-    allowed = set(config.get("tags", []))
+    soup = BeautifulSoup(html, "html.parser")
    good_soup = BeautifulSoup(html, "html.parser")
-    def has_allowed_child(node: Tag) -> bool:
+    original_head = soup.find("head")
-        # check if shit is allowed
+    original_body = soup.find("body")
        for child in node.descendants:
            if isinstance(child, Tag) and child.name.lower() in allowed:
                return True
        return False
    head_html = str(original_head) if original_head else ""
    body_html = str(original_body) if original_body else html
-    for child in list(good_soup.find_all()):
+    head_tags = {"title", "meta", "link", "base", "style"}
-        name = child.name.lower()
+    head_allowed = set(config.get("tags", [])) & head_tags
        if name in allowed:
            continue
        if has_allowed_child(child):
            child.unwrap()
        else:
            child.decompose()
-    bs_parsed = str(good_soup)
+    cleaned_head_content = bleach.clean(
        head_html,
        tags=head_allowed,
        attributes=config.get("attributes"),
        protocols=config.get("protocols"),
        strip=config.get("strip"),
    )
    cleaned_body_content = bleach.clean(
        body_html,
        tags=config.get("tags"),
        attributes=config.get("attributes"),
        protocols=config.get("protocols"),
        strip=config.get("strip"),
    )
    pretty_body = BeautifulSoup(cleaned_body_content, "html.parser").prettify()
    pretty_head = BeautifulSoup(cleaned_head_content, "html.parser").prettify()
    logger.log_debug(
-        "called sanitizer! using this as args: \n" + 
+        # very ugly string concationation for logging
-        str(bs_parsed) +
+        "parser args: "     + str(
-        str(config["tags"]) + "\n" +
+            "Tags: "        + str(config.get("tags")) + "\n" + 
-        str(config["protocols"]) + "\n" +
+            "Attributes: "  + str(config.get("attributes")) + "\n" +
-        str(config["strip"]) 
+            "Protocols: "   + str(config.get("protocols")) + "\n" +
            "Strip: "       + str(config.get("strip")) + "\n"
            ) + 
        "input html: " + str(html) + 
        "clean head: " + str(pretty_head) + 
        "clean body: " + str(pretty_body) 
    )
-    return bleach.clean(
+    return f"""
-        bs_parsed,
+<!DOCTYPE html>
-        tags=config["tags"],
+<html>
-        attributes=config["attributes"],
+<!-- sanigi-html parsed @ {datetime.datetime.now()} (version: python) -->
-        protocols=config["protocols"],
+<head>{pretty_head}</head>
-        strip=config["strip"],
+<body>{pretty_body}</body>
-    )
+</html>
 """
 def _wrap_document(content: str) -> str:
    logger.log_debug("Content:" + content)
    soup = BeautifulSoup(content, "html.parser")
    existing_head = soup.find("head")
    logger.log_debug("Existing Head?:" + str(existing_head))
    existing_body = soup.find("body")
    logger.log_debug("Existing Body?:" + str(existing_body))
    head_content = str(existing_head) if existing_head else "<head></head>"
    body_content = str(existing_body) if existing_body else f"<body>{content}</body>"
    return f"""
 <!DOCTYPE html>
 <html>
 <!-- sanigi-html parsed @ {datetime.datetime.now()} -->
 {head_content}
 {body_content}
 </html>
    """
Author	SHA1	Message	Date
rattatwinko	5c6746d00a	Merge pull request 'development' (#1 ) from development into main Reviewed-on: #1	2026-04-23 06:28:06 +00:00
rattatwinko	e5cf74245a	add link and logging	2026-04-23 08:25:52 +02:00
rattatwinko	bd935c62f5	fixes to HTML head	2026-04-22 20:39:59 +02:00
rattatwinko	10a09db853	remove out.html	2026-04-20 12:43:40 +02:00
rattatwinko	1afc450711	implement configuration loading with json parser, implement some new argument for it. ship with example json configuration file. add: method to add doctype/html/head/body to document. see examples in todo! with progamm usage	2026-04-20 12:42:54 +02:00