Compare commits

..

4 Commits
main ... master

Author SHA1 Message Date
decentral1se
4964790762
follow title in upstream README 2022-02-04 13:01:12 +01:00
decentral1se
9ef68194db
caps that 2022-02-04 13:00:52 +01:00
decentral1se
1418feeaa8
fix all my typos 2022-02-04 13:00:14 +01:00
decentral1se
06cf1f5bdd
more uses 2022-02-04 12:59:12 +01:00
10 changed files with 286 additions and 358 deletions

View File

@ -8,15 +8,6 @@ index functions featured in several web servers. It works by traversing the
file system and directory hierarchy to automatically list all the files in the
directory and providing them with html classes and tags for easy styling.
## This distribusi fork powers distribusi-verse
If you want to run distribusi on its own go here: [original](https://git.vvvvvvaria.org/varia/distribusi)
This version is used to power [distribusi-verse](https://git.vvvvvvaria.org/Toolsheds/distribusi-verse) which folds this tool in a webapplication.
To make this work a little easier this version has certain options removed that are handy
and available in the standalone version.
For other distribusi like softwares also checkout [distribusi-go](https://git.coopcloud.tech/decentral1se/distribusi-go)
## Requirements
While a Pip install will pull in Python dependencies, you might need system
@ -27,6 +18,13 @@ packages:
* [github.com/threatstack/libmagic](https://github.com/threatstack/libmagic)
* [pillow.readthedocs.io](https://pillow.readthedocs.io/en/5.3.x/installation.html#external-libraries)
### Optional requirements
If you wish to use the `--caption` flag to add image captions read from EXIF comment metadata you will need a utility called `exiftool`.
You can install it via your package manager. For other options please consult the website: [https://www.sno.phy.queensu.ca/~phil/exiftool/](https://www.sno.phy.queensu.ca/~phil/exiftool/)
## Install It
```bash
@ -64,7 +62,7 @@ Create a quick gallery for the web:
$ distribusi -d /path/to/my/photos -t
```
This creates an `index.html` with accompanying `_thumbnail.jpg` files
This creates an `index.html` with `base64` encoded thumbnails.
Generate verbose output:
@ -94,3 +92,39 @@ The contribution consisted of setting up distribusi.ruruhuis.nl (distribusi is b
> "Individuals collecting fragments each have their own folder. When they put a story, picture, audio, or video file inside a folder, each fragment is assigned a serial number, and gets accumulated in the fragments of hospitality website.The fragments connect us. We hope that we can choose the “nature and attitude” of the medium that mediates our connection."
[https://fragments1444.ink/about.html](source)
### Distribusi verse
> An attempt to make distribusi into a web interface that can be operated remotely without any knowlegde of CLI. Trying to somehow combine the ideas of distribusi with the ideas of a [tildeverse](https://tildeverse.org/) or [Tilde club](https://tilde.club), but also be neither of these ideas. This project is made for Autonomous Practices at the WDKA in Rotterdam.
See [`crunk/distribusi-verse`](https://git.vvvvvvaria.org/crunk/distribusi-verse) for more.
### `distribusi-go`
Inspired a re-implementation in Go, see [`decentral1se/distribusi-go`](https://git.vvvvvvaria.org/decentral1se/distribusi-go) for more.
## Change It
You'll need to get a copy of the repository and then do an [editable] install:
[editable]: https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode
```bash
$ git clone https://git.vvvvvvaria.org/varia/distribusi.git && cd distribusi
$ python3 -m venv .venv && source .venv/bin/activate
$ pip install -e .
```
You're then ready to make your changes and experiment with them.
## Release It
You'll need a [PyPi](https://pypi.org/) account and to be added as a maintainer.
Please ask around @ Varia for who has PyPi access.
```
$ # ... change the version number in setup.py ... #
$ pip install twine wheel
$ make publish
```

View File

@ -6,79 +6,83 @@ from distribusi.distribusi import distribusify
def build_argparser():
parser = argparse.ArgumentParser(
"""
"""
distribusi is a content management system for the web that produces static
index pages based on folders in the files system. It is inspired by the
automatic index functions featured in several popular web servers.
distribusi works by traversing the file system and directory hierarchy to
automatically list all the files in the directory, detect the file types
and providing them with relevant html classes and tags for easy styling.
"""
""")
parser.add_argument(
'-d', '--directory', help="Select which directory to distribute", default="."
)
parser.add_argument(
"-d",
"--directory",
help="Select which directory to distribute",
default=".",
'-s', '--style', help="Select a CSS style sheet to include"
)
parser.add_argument(
"-s", "--style", help="Select a CSS style sheet to include"
'-v', '--verbose', help="Print verbose debug output", action="store_true"
)
parser.add_argument(
"-v",
"--verbose",
help="Print verbose debug output",
action="store_true",
)
parser.add_argument(
"-t",
"--thumbnail",
'-t',
'--thumbnail',
help="Generate 450x450 thumbnails for images",
action="store_true",
)
parser.add_argument(
"-a",
"--alttexts",
help="Adds file alttext based on same named files",
'-n',
'--no-template',
help="Don't use the template to output html",
action="store_true",
)
parser.add_argument(
"-r",
"--remove-index",
'-nf',
'--no-filenames',
help="Don't add file names to listing",
action="store_true",
)
parser.add_argument(
'-c',
'--captions',
help="Adds image captions based on EXIF metadata, requires 'exiftool'",
action="store_true",
)
parser.add_argument(
'-r',
'--remove-index',
help="Recursively removes all instances of index.html that have been previously made by distribusi",
action="store_true",
)
action="store_true")
parser.add_argument(
"-e",
"--exclude-directory",
'-e',
'--exclude-directory',
help="Exclude one or multiple directories from indexing",
nargs="*",
metavar="DIR",
)
metavar='DIR')
parser.add_argument(
"-f",
"--force",
'-f',
'--force',
help="Force whether distribusi overwrites or removes instances of index.html not generated by distribusi, use at own risk!",
action="store_true",
)
action="store_true")
parser.add_argument(
"--no-hidden", help="Exclude hidden directories", action="store_true"
)
'--no-hidden',
help="Exclude hidden directories",
action="store_true")
parser.add_argument(
"--menu-with-index",
'--menu-with-index',
help="Append index.html to menu items to aid navigation",
action="store_true",
)
action="store_true")
return parser

View File

@ -1,85 +1,75 @@
import base64
import os
import subprocess
from io import BytesIO
import magic
from PIL import Image
from distribusi.templates.page_template import html_footer, html_head
from distribusi.templates.image_templates import (
image_no_description,
image_with_description,
image_with_alttext,
image_full_figure,
)
from distribusi.page_template import html_footer, html_head
from distribusi.mappings import CODE_TYPES, FILE_TYPES, SUB_TYPES
MIME_TYPE = magic.Magic(mime=True)
def add_alttext(full_path_image):
def caption(image):
try:
image_filename_no_ext = os.path.splitext(full_path_image)[0]
alttext_filename = f"{image_filename_no_ext}_alttext.txt"
return _read_matching_text_file(
image_filename_no_ext, alttext_filename
)
process = subprocess.Popen(
['exiftool', '-Comment', image], stdout=subprocess.PIPE)
out, err = process.communicate()
except Exception as e:
print(f"exception {e} raised while making alttext")
return
def add_description(full_path_image):
print(e)
print('Do you have exiftool installed?')
try:
image_filename_no_ext = os.path.splitext(full_path_image)[0]
description_filename = f"{image_filename_no_ext}_dv_description.txt"
return _read_matching_text_file(
image_filename_no_ext, description_filename
)
caption = out.decode("utf-8").split(": ", 1)[1]
except Exception as e:
print(f"exception {e} raised while adding description")
return
caption = ''
print(e)
return caption
def _read_matching_text_file(image_filename_no_ext, filename):
if not os.path.isfile(filename):
return
print(f"{image_filename_no_ext} has {filename}")
with open(filename, "r") as text_file:
return text_file.read()
def thumbnail(full_path_image, name, args):
if full_path_image.endswith("_thumbnail.jpg"):
return
def thumbnail(image, name, args):
try:
size = (450, 450)
thumbnail_image = Image.open(full_path_image)
thumbnail_image.thumbnail(size)
if thumbnail_image.mode == "RGBA":
bg = Image.new("RGBA", thumbnail_image.size, (255, 255, 255))
composite = Image.alpha_composite(bg, thumbnail_image)
thumbnail_image = composite.convert("RGB")
image_filename_no_ext = os.path.splitext(full_path_image)[0]
thumbnail_filename = f"{image_filename_no_ext}_thumbnail.jpg"
thumbnail_image.save(thumbnail_filename, format="JPEG")
return os.path.basename(thumbnail_filename)
im = Image.open(image)
im.thumbnail(size)
if (im.mode == 'RGBA'):
bg = Image.new('RGBA', im.size, (255,255,255))
composite = Image.alpha_composite(bg, im)
im=composite.convert('RGB')
output = BytesIO()
im.save(output, format='JPEG')
im_data = output.getvalue()
data_url = base64.b64encode(im_data).decode()
if args.captions:
cap = caption(image)
else:
cap = name
return (
"<figure><a href='{}'><img class='thumbnail' src='data:image/jpg;base64,{}'></a><figcaption>{}</figcaption></figure>"
).format(name, data_url, cap)
except Exception as e:
print("Thumbnailer:", e)
return
print('Thumbnailer:', e)
return "<figure><a href='{}'><img src='{}'></a><figcaption>{}</figcaption></figure>".format(name, name, name)
def format_div(args, type_, subtype, tag, name):
id_name = name.split(".")[0].replace(" ", "_")
filename = f'<span class="filename">{name}</span>'
def div(args, type_, subtype, tag, name):
id_name = name.split('.')[0].replace(' ', '_')
if args.no_filenames:
filename = ''
else:
filename = '<span class="filename">{}</span>'.format(name)
if "image" in type_:
if 'image' in type_:
html = '<div id="{}" class="{}">{}</div>'
elif "pdf" in subtype:
html = '<div id="{}" class="{}">{}' + filename + "</div>"
elif "dir" in type_ or "html" in subtype or "unkown-file" in subtype:
elif 'pdf' in subtype:
html = '<div id="{}" class="{}">{}' + filename + '</div>'
elif 'dir' in type_ or 'html' in subtype or 'unkown-file' in subtype:
html = '<div id="{}" class="{}">{}</div>'
else:
html = '<div id="{}" class="{}">{}' + filename + "</div>"
html = '<div id="{}" class="{}">{}' + filename + '</div>'
return html.format(id_name, subtype, tag)
@ -88,188 +78,142 @@ def check_distribusi_index(args, index):
"""
check whether a index.html file is generated by distribusi
"""
if not args.force:
with open(index, "r") as f:
with open(index, 'r') as f:
if '<meta name="generator" content="distribusi" />' in f.read():
return True
else:
if args.verbose:
print(index, "not generated by distribusi, skipping")
print(index, 'not generated by distribusi, skipping')
return False
elif args.force:
return True
def write_index(args,index, html, html_head, html_footer):
with open(index, 'w') as f:
if not args.no_template:
if args.style:
fs = open(args.style, "r")
style = fs.read()
styled_html_head = html_head % style
else:
styled_html_head = html_head % ''
f.write(styled_html_head)
def write_index_html(args, index, html):
with open(index, "w") as index_file:
styled_html_head = html_head.format(cssfile=args.style)
index_file.write(styled_html_head)
for line in html:
index_file.write(line + "\n")
index_file.write(html_footer)
f.write(line + '\n')
def handle_text_files(name, full_path, subtype):
if name.endswith(".html") or subtype == "html":
subtype = "html"
# what types of text files to expand
tag = '<section id="{}">{}</section>'.format(
name, open(full_path).read()
)
elif subtype in CODE_TYPES or name.endswith(".txt"):
# if the plain text is code,
# which types do we wrap in pre-tags?
tag = "<pre>" + open(full_path).read() + "</pre>"
else:
subtype = subtype + " unkown-file"
tag = "<a href='{}'>{}</a>"
# a = FILE_TYPES[type_]
return subtype, tag
def handle_image_files(name, full_path, args):
if args.thumbnail:
thumbnail_filename = thumbnail(full_path, name, args)
if thumbnail_filename is None:
return
image_alttext = add_alttext(full_path)
image_description = add_description(full_path)
if not image_alttext and not image_description:
return image_no_description.format(
name=name, thumbnail_filename=thumbnail_filename
)
if not image_alttext:
return image_with_description.format(
name=name,
thumbnail_filename=thumbnail_filename,
image_description=image_description,
)
if not image_description:
return image_with_alttext.format(
name=name,
thumbnail_filename=thumbnail_filename,
image_alttext=image_alttext,
)
return image_full_figure.format(
name=name,
thumbnail_filename=thumbnail_filename,
image_alttext=image_alttext,
image_description=image_description,
)
return FILE_TYPES["image"].format(name, image_alttext)
def remove_index_html(root, files, args):
index = os.path.join(root, "index.html")
if "index.html" in files:
try:
if check_distribusi_index(args, index):
if args.verbose:
print("Removing index.html from", root)
os.remove(index)
except Exception as e:
print(e)
return
def remove_hidden_files_and_folders(dirs, files):
dirs = list(filter(lambda d: not d.startswith("."), dirs))
files = list(filter(lambda f: not f.startswith("."), files))
return dirs, files
def is_skipped_file(name, args):
if "index.html" in name:
return True
if name.endswith("_thumbnail.jpg"):
return True
if name.endswith("_alttext.txt"):
return True
if name.endswith("_dv_description.txt"):
return True
if args.style in name:
return True
if not args.no_template:
f.write(html_footer)
def distribusify(args, directory): # noqa
for root, dirs, files in os.walk(directory):
html = []
if args.exclude_directory:
if args.verbose:
print(
"Excluding directory:", ", ".join(args.exclude_directory)
)
print('Excluding directory:', ", ".join(args.exclude_directory))
dirs[:] = [d for d in dirs if d not in args.exclude_directory]
if args.no_hidden:
dirs, files = remove_hidden_files_and_folders(dirs, files)
dirs = list(filter(lambda d: not d.startswith('.'), dirs))
files = list(filter(lambda f: not f.startswith('.'), files))
if args.remove_index:
remove_index_html(root, files, args)
dirs.sort()
files.sort()
if args.verbose:
print("Generating directory listing for", root)
for name in sorted(files):
if is_skipped_file(name, args):
continue
full_path = os.path.join(root, name)
mime = MIME_TYPE.from_file(full_path)
type_, subtype = mime.split("/")
alttext = name
if not args.remove_index:
html = []
if args.verbose:
print("Found", name, "as", mime)
if type_ in FILE_TYPES:
match type_:
case "text":
subtype, tag = handle_text_files(
name, full_path, subtype
)
case "image":
tag = handle_image_files(name, full_path, args)
if tag is None:
continue
case _:
tag = FILE_TYPES[type_].format(name, alttext)
print('Generating directory listing for', root)
if subtype in SUB_TYPES:
tag = SUB_TYPES[subtype]
for name in sorted(files):
if type_ not in FILE_TYPES and subtype not in SUB_TYPES:
# catch exceptions not yet defined in FILE_TYPES or SUB_TYPES
tag = "<a href='{}'>{}</a>"
if args.verbose:
message = (
"not in list of file types, adding as plain href: \n"
)
print(type_, subtype, message, name)
subtype = subtype + " unkown-file"
if 'index.html' not in name:
full_path = os.path.join(root, name)
mime = MIME_TYPE.from_file(full_path)
# example: MIME plain/text becomes 'type' plain 'subtype' text
type_, subtype = mime.split('/')
tag = tag.replace("{}", name)
html.append(format_div(args, type_, subtype, tag, name))
caption = name
if root != directory:
if args.menu_with_index:
html.append('<a href="../index.html">../</a>')
else:
html.append('<a href="../">../</a>')
if args.verbose:
print('Found', name, 'as', mime)
for name in sorted(dirs):
if args.menu_with_index:
tag = "<a href='{}/index.html'>{}</a>".replace("{}", name)
else:
tag = "<a href='{}'>{}/</a>".replace("{}", name)
if type_ in FILE_TYPES:
a = FILE_TYPES[type_].format(name, caption)
html.insert(0, format_div(args, "dir", "dir", tag, "folder"))
# expansion for different kind of text files
if type_ == 'text':
if name.endswith('.html') or subtype == 'html':
subtype = 'html'
# what types of text files to expand
a = '<section id="{}">{}</section>'.format(name, open(full_path).read())
elif subtype in CODE_TYPES or name.endswith('.txt'):
# if the plain text is code,
# which types do we wrap in pre-tags?
a = "<pre>" + open(full_path).read() + "</pre>"
else:
subtype = subtype+' unkown-file'
a = "<a href='{}'>{}</a>"
# a = FILE_TYPES[type_]
index = os.path.join(root, "index.html")
write_index_html(args, index, html)
if type_ == 'image':
if args.thumbnail:
a = thumbnail(full_path, name, args)
if args.no_filenames:
caption = ""
a = FILE_TYPES[type_].format(name, caption)
if args.captions:
caption = caption(full_path)
a = FILE_TYPES[type_].format(name, caption)
if subtype in SUB_TYPES:
a = SUB_TYPES[subtype]
if type_ not in FILE_TYPES and subtype not in SUB_TYPES:
# catch exceptions not yet defined in FILE_TYPES or SUB_TYPES
a = "<a href='{}'>{}</a>"
if args.verbose:
message = 'not in list of file types, adding as plain href: \n'
print(type_, subtype, message, name)
subtype = subtype + ' unkown-file'
a = a.replace('{}', name)
html.append(div(args, type_, subtype, a, name))
if root != directory:
if args.menu_with_index:
html.append('<a href="../index.html">../</a>')
else:
html.append('<a href="../">../</a>')
for name in dirs:
if args.menu_with_index:
a = "<a href='{}/index.html'>{}</a>".replace('{}', name)
else:
a = "<a href='{}'>{}/</a>".replace('{}', name)
html.insert(0, div(args, 'dir', 'dir', a, 'folder'))
index = os.path.join(root, 'index.html')
if os.path.exists(index):
if check_distribusi_index(args, index):
write_index(args,index,html, html_head, html_footer)
elif not os.path.exists(index):
write_index(args,index,html, html_head, html_footer)
if args.remove_index:
index = os.path.join(root, 'index.html')
if 'index.html' in files:
try:
if check_distribusi_index(args, index):
if args.verbose:
print('Removing index.html from', root)
os.remove(index)
except Exception as e:
print(e)

View File

@ -1,15 +1,14 @@
CODE_TYPES = ["x-c", "x-shellscript", "x-python"]
CODE_TYPES = ['x-c', 'x-shellscript', 'x-python']
FILE_TYPES = {
"image": '<img class="image" src="{}" alt="{}">',
"text": '<a href="{}" class="text">{}</a>',
"video": ("<video controls>" '<source src="{}"></video>'),
"audio": ('<audio controls class="audio">' '<source src="{}"></audio>'),
'image': '<figure><img class="image" src="{}"><figcaption>{}</figcaption></figure>',
'text': '<a href="{}" class="text">{}</a>',
'video': ('<video controls>' '<source src="{}"></video>'),
'audio': ('<audio controls class="audio">' '<source src="{}"></audio>'),
}
SUB_TYPES = {
"pdf": (
'<object data="{}" class="pdf" type="application/pdf">'
'<embed src="{}" type="application/pdf" /></object>'
)
'pdf': (
'<object data="{}" class="pdf" type="application/pdf">'
'<embed src="{}" type="application/pdf" /></object>')
}

View File

@ -0,0 +1,25 @@
html_head = """
<!DOCTYPE html>
<html lang="en">
<head>
<!-- Generated with distribusi https://git.vvvvvvaria.org/varia/distribusi -->
<meta name="generator" content="distribusi" />
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<style>
.image{max-width: 100%%;}
.pdf object{width:640px;height: 640px;}
.dir::before{content:"📁 ";font-size:18px;}
.filename{display:block;font-family:mono;}
.unkown-file::before{content:"📄 ";font-size:18px;}
div{max-width: 640px;display:inline-block;vertical-align:top;margin:1em;padding:1em;}
video {width:640px;max-height:640px;}
%s
</style>
</head>
<body>
"""
html_footer = """
</body>
</html>
"""

View File

@ -1,4 +0,0 @@
image_no_description = '<a href="{name}"><img class="thumbnail" src="{thumbnail_filename}" alt="{name}"></a>'
image_with_description = '<figure><a href="{name}"><img class="thumbnail" src="{thumbnail_filename}"></a><figcaption>{image_description}</figcaption></figure>'
image_with_alttext = '<a href="{name}"><img class="thumbnail" src="{thumbnail_filename}" alt="{image_alttext}"></a>'
image_full_figure = '<figure><a href="{name}"><img class="thumbnail" src="{thumbnail_filename}" alt="{image_alttext}"></a><figcaption>{image_description}</figcaption></figure>'

View File

@ -1,16 +0,0 @@
html_head = '''\
<!DOCTYPE html>
<html lang="en">
<head>
<!-- Generated with distribusi https://git.vvvvvvaria.org/crunk/distribusi -->
<meta name="generator" content="distribusi" />
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<link rel="stylesheet" href="{cssfile}" crossorigin="anonymous">
</head>
<body>
'''
html_footer ='''\
</body>
</html>
'''

View File

@ -7,53 +7,3 @@ build-backend = "setuptools.build_meta"
[tool.black]
skip-string-normalization = true
[tool.ruff]
line-length = 79
target-version = "py311"
#include = '\.pyi?$'
exclude = [
".bzr",
".direnv",
".eggs",
".git",
".git-rewrite",
".hg",
".ipynb_checkpoints",
".mypy_cache",
".nox",
".pants.d",
".pyenv",
".pytest_cache",
".pytype",
".ruff_cache",
".svn",
".tox",
".venv",
".vscode",
"__pypackages__",
"_build",
"buck-out",
"build",
"dist",
"node_modules",
"site-packages",
"venv",
]
[tool.ruff.lint]
# Enable Pyflakes (`F`) and a subset of the pycodestyle (`E`) codes by default.
select = ["E4", "E7", "E9", "F"]
ignore = []
# Allow fix for all enabled rules (when `--fix`) is provided.
fixable = ["ALL"]
unfixable = []
# Allow unused variables when underscore-prefixed.
dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
[tool.ruff.format]
quote-style = "double"
indent-style = "space"
docstring-code-format = true
line-ending = "auto"
skip-magic-trailing-comma = false

View File

@ -1,36 +1,28 @@
from setuptools import find_packages, setup
dependencies = [
"pillow >= 10.3.0",
"python-magic >= 0.4.15, < 1.0",
]
dependencies = ['pillow >= 6.1.0, < 7.0', 'python-magic >= 0.4.15, < 1.0']
with open("README.md", "r") as handle:
with open('README.md', 'r') as handle:
long_description = handle.read()
setup(
name="distribusi",
version="0.0.12",
url="https://git.vvvvvvaria.org/crunk/distribusi",
license="GPLv3",
author="Varia",
name='distribusi',
version='0.0.10',
url='https://git.vvvvvvaria.org/varia/distribusi',
license='GPLv3',
author='Varia',
description=(
"Distribusi is a content management system for "
"the web that produces static pages based on "
"the file system."
'Distribusi is a content management system for '
'the web that produces static pages based on '
'the file system.'
),
long_description=long_description,
long_description_content_type="text/markdown",
packages=find_packages(exclude=["tests"]),
long_description_content_type='text/markdown',
packages=find_packages(exclude=['tests']),
include_package_data=True,
zip_safe=False,
platforms="any",
platforms='any',
install_requires=dependencies,
entry_points={
"console_scripts": ["distribusi = distribusi.cli:cli_entrypoint"]
},
classifiers=[
"Programming Language :: Python :: 3",
"Environment :: Console",
],
entry_points={'console_scripts': ['distribusi = distribusi.cli:cli_entrypoint']},
classifiers=['Programming Language :: Python :: 3', 'Environment :: Console'],
)