Browse Source

Use poetry, run formatters

main
decentral1se 6 months ago
parent
commit
e66792ec3e
No known key found for this signature in database GPG Key ID: 5E2EF5A63E3718CC
  1. 0
      LICENSE
  2. 1
      MANIFEST.in
  3. 19
      Makefile
  4. 93
      README.md
  5. 97
      bin/etherpump
  6. 102
      etherpump/__init__.py
  7. 8
      etherpump/commands/common.py
  8. 14
      etherpump/commands/creatediffhtml.py
  9. 10
      etherpump/commands/deletepad.py
  10. 26
      etherpump/commands/dumpcsv.py
  11. 12
      etherpump/commands/gethtml.py
  12. 12
      etherpump/commands/gettext.py
  13. 26
      etherpump/commands/index.py
  14. 6
      etherpump/commands/list.py
  15. 8
      etherpump/commands/listauthors.py
  16. 36
      etherpump/commands/publication.py
  17. 60
      etherpump/commands/pull.py
  18. 8
      etherpump/commands/revisionscount.py
  19. 20
      etherpump/commands/sethtml.py
  20. 18
      etherpump/commands/settext.py
  21. 2
      etherpump/commands/showmeta.py
  22. 8
      etherpump/commands/status.py
  23. 12
      padinfo.sample.json
  24. 786
      poetry.lock
  25. 50
      pyproject.toml
  26. 9
      setup.cfg
  27. 59
      setup.py

0
LICENSE.txt → LICENSE

1
MANIFEST.in

@ -1 +0,0 @@
include etherpump/data/templates/*

19
Makefile

@ -1,13 +1,14 @@
SOURCE_DIRS := bin/ etherpump/
publish:
@rm -rf dist
@python setup.py bdist_wheel
@twine upload dist/*
default: style
format:
@black $(SOURCE_DIRS)
@isort -rc $(SOURCE_DIRS)
@poetry run black etherpump
sort:
@poetry run isort etherpump
lint:
@flake8 $(SOURCE_DIRS)
@poetry run flake8 etherpump
style: format sort lint
.PHONY: style format sort lint

93
README.md

@ -1,24 +1,22 @@
etherpump
=========
# etherpump
[![PyPI version](https://badge.fury.io/py/etherpump.svg)](https://badge.fury.io/py/etherpump)
[![GPL license](https://img.shields.io/badge/license-GPL-brightgreen.svg)](https://git.vvvvvvaria.org/varia/etherpump/src/branch/master/LICENSE.txt)
*Pumping text from etherpads into publications*
_Pumping text from etherpads into publications_
A command-line utility that extends the multi writing and publishing functionalities of the [etherpad](http://etherpad.org/) by exporting the pads in multiple formats.
Many pads, many networks
------------------------
## Many pads, many networks
*Etherpump* is a friendly fork of [*etherdump*](https://gitlab.constantvzw.org/aa/etherdump), a command line tool written by [Michael Murtaugh](http://automatist.org/) that converts etherpad pages to files. This fork is made out of curiosities in the tool, a wish to study it and shared sparks of enthusiasm to use it in different situations within Varia.
_Etherpump_ is a friendly fork of [_etherdump_](https://gitlab.constantvzw.org/aa/etherdump), a command line tool written by [Michael Murtaugh](http://automatist.org/) that converts etherpad pages to files. This fork is made out of curiosities in the tool, a wish to study it and shared sparks of enthusiasm to use it in different situations within Varia.
Etherpump is a stretched version of etherdump. It is a playground in which we would like to add features to the initial tool that diffuse actions of *dumping* into *pumping*. So most of all, etherpump is a work-in-progress, exploring potential uses of etherpads to edit, structure and publish various types of content.
Etherpump is a stretched version of etherdump. It is a playground in which we would like to add features to the initial tool that diffuse actions of _dumping_ into _pumping_. So most of all, etherpump is a work-in-progress, exploring potential uses of etherpads to edit, structure and publish various types of content.
Added features are:
* opt-in publishing with the `__PUBLISH__` magic word
* the `publication` command, that listens to custom magic words such as `__RELEARN__`
- opt-in publishing with the `__PUBLISH__` magic word
- the `publication` command, that listens to custom magic words such as `__RELEARN__`
See the [Change log / notes ](#change-log--notes) section for further changes.
@ -28,8 +26,11 @@ We started to get to know etherpump through various editions of Relearn and/or t
After installing etherpump on the Varia server, we collectively decided to not want to publish pads by default. Discussions in the group around the use of etherpads, privacy and ideas of what publishing means, led to a need to have etherpump only start the indexing work after it recognizes a `__PUBLISH__` marker on a pad. We decided to work on a `__PUBLISH__ vs. __NOPUBLISH__` branch of etherdump, which we now fork into **etherpump**.
Change log / notes
==================
# Change log / notes
**October 2020**
Use the more friendly packaging tool [Poetry](https://python-poetry.org/) for publishing.
**January 2020**
@ -56,7 +57,7 @@ Started with the [experimental library API](#library-api-example).
**September 2019**
Forking *etherpump* into *etherpump*.
Forking _etherpump_ into _etherpump_.
<https://git.vvvvvvaria.org/varia/etherpump>
@ -64,25 +65,25 @@ Migrating the source code to Python 3.
Integrate PyPi publishing with setuptools.
-----
---
**May - September 2019**
etherpump is used to produce the *Ruminating Relearn* section of the Network Of One's Own 2 (NOOO2) publication.
etherpump is used to produce the _Ruminating Relearn_ section of the Network Of One's Own 2 (NOOO2) publication.
A new command is added to make a web publication, based on the custom magic word `__RELEARN__`.
-----
---
**June 2019**
Multiple conversations around etherpump emerged during Relearn Curved in Varia, Rotterdam.
Including the idea of executable pads (*etherhooks*), custom magic words, a federated snippet protocol (*etherstekje*) and more.
Including the idea of executable pads (_etherhooks_), custom magic words, a federated snippet protocol (_etherstekje_) and more.
<https://varia.zone/relearn-2019.html>
-----
---
**April 2019**
@ -90,30 +91,27 @@ Installation of etherpump on the Varia server.
<https://etherpump.vvvvvvaria.org/>
-----
---
**March 2019**
The `__PUBLISH__ vs. __NOPUBLISH__` was added to the etherpump repository by *decentral1se*.
The `__PUBLISH__ vs. __NOPUBLISH__` was added to the etherpump repository by _decentral1se_.
<https://gitlab.constantvzw.org/aa/etherpump/issues/3>
-----
---
Originally designed for use at: [Constant](http://etherdump.constantvzw.org/).
More notes can be found in the [git repository of etherdump](https://gitlab.constantvzw.org/aa/etherdump).
Install etherpump
=================
# Install etherpump
`$ pip install etherpump`
Etherpump only supports Python 3.
Command-line example
--------------------
## Command-line example
```
$ mkdir mydump
@ -133,8 +131,7 @@ The APIKEY is the contents of the file APIKEY.txt in the etherpad folder.
The settings are placed in a file called `.etherpump/settings.json` and are used (by default) by future commands.
Library API Example
-------------------
## Library API Example
Etherpump can be used as a library.
@ -145,8 +142,7 @@ All commands can be imported and run programmatically.
>>> pull(['--all', '--publish-opt-in', '--publish', '__PUB_CLUB__'])
```
Subcommands
----------
## Subcommands
To see all available subcommands, run:
@ -156,41 +152,40 @@ For help on each individual subcommand, run:
`$ etherpump revisionscount --help`
Publishing
----------
## Publishing
Please use ["semver"](https://semver.org/) conventions for versions.
Here are the steps to follow (e.g. for a `0.1.3` release):
* `pip install twine`
* Bump the version number in `etherpump/__init__.py` following ["semver"](https://semver.org/) conventions
* Run `make publish`
- Change the version number in the `etherpump/__init__.py` `__VERSION__` to `0.1.3`
- Change the version number in the `pyproject.toml` `version` field to `0.1.3`
- `git add . && git commit -m "Publish new 0.1.3 version" && git tag 0.1.3 && git push --tags`
- Run `poetry publish --build`
You should have a [PyPi](https://pypi.org/) account and be added as an owner/maintainer on the [etherpump package](https://pypi.org/project/etherpump/).
Maintenance utilities
---------------------
## Maintenance utilities
Tools to help things stay tidy over time.
```bash
$ pip install flake8 isort black
$ make format
$ make lint
$ make
```
Please see the following links for further reading:
* http://flake8.pycqa.org
* https://isort.readthedocs.io
* https://black.readthedocs.io
- [flake8](http://flake8.pycqa.org)
- [isort](https://isort.readthedocs.io)
- [black](https://black.readthedocs.io)
Keeping track of Etherpad-lite
------------------------------
## Keeping track of Etherpad-lite
* [Etherpad-lite API documentation](https://etherpad.org/doc/v1.7.5/)
* [Etherpad-lite releases](https://github.com/ether/etherpad-lite/releases)
- [Etherpad-lite API documentation](https://etherpad.org/doc/v1.7.5/)
- [Etherpad-lite releases](https://github.com/ether/etherpad-lite/releases)
License
=======
# License
GNU AFFERO GENERAL PUBLIC LICENSE, Version 3.
See [LICENSE.txt](./LICENSE.txt).
See [LICENSE](./LICENSE).

97
bin/etherpump

@ -1,97 +0,0 @@
#!/usr/bin/env python3
import sys
from etherpump import __VERSION__
def subcommands():
"""List all sub-commands for the `--help` output."""
output = []
subcommands = [
'creatediffhtml',
'deletepad',
'dumpcsv',
'gethtml',
'gettext',
'index',
'init',
'list',
'listauthors',
'publication',
'pull',
'revisionscount',
'sethtml',
'settext',
'showmeta',
]
for subcommand in subcommands:
try:
# http://stackoverflow.com/questions/301134/dynamic-module-import-in-python
doc = __import__(
"etherpump.commands.%s" % subcommand,
fromlist=["etherdump.commands"],
).__doc__
except ModuleNotFoundError:
doc = ""
output.append(f' {subcommand}: {doc}')
output.sort()
return '\n'.join(output)
usage = """
_
| |
_ _|_ | | _ ,_ _ _ _ _ _
|/ | |/ \ |/ / | |/ \_| | / |/ |/ | |/ \_
|__/|_/| |_/|__/ |_/|__/ \_/|_/ | | |_/|__/
/| /|
\| \|
Usage:
etherpump CMD
where CMD could be:
{}
For more information on each command try:
etherpump CMD --help
""".format(
subcommands()
)
try:
cmd = sys.argv[1]
if cmd.startswith("-"):
args = sys.argv
else:
args = sys.argv[2:]
if len(sys.argv) < 3:
if any(arg in args for arg in ['--help', '-h']):
print(usage)
sys.exit(0)
elif any(arg in args for arg in ['--version', '-v']):
print('etherpump {}'.format(__VERSION__))
sys.exit(0)
except IndexError:
print(usage)
sys.exit(0)
try:
# http://stackoverflow.com/questions/301134/dynamic-module-import-in-python
cmdmod = __import__(
"etherpump.commands.%s" % cmd, fromlist=["etherdump.commands"]
)
cmdmod.main(args)
except ImportError as e:
print("Error performing command '{0}'\n(python said: {1})\n".format(cmd, e))
print(usage)

102
etherpump/__init__.py

@ -1,5 +1,105 @@
#!/usr/bin/env python3
import os
import sys
DATAPATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), "data")
__VERSION__ = '0.0.13'
__VERSION__ = "0.0.13"
def subcommands():
"""List all sub-commands for the `--help` output."""
output = []
subcommands = [
"creatediffhtml",
"deletepad",
"dumpcsv",
"gethtml",
"gettext",
"index",
"init",
"list",
"listauthors",
"publication",
"pull",
"revisionscount",
"sethtml",
"settext",
"showmeta",
]
for subcommand in subcommands:
try:
# http://stackoverflow.com/questions/301134/dynamic-module-import-in-python
doc = __import__(
"etherpump.commands.%s" % subcommand,
fromlist=["etherdump.commands"],
).__doc__
except ModuleNotFoundError:
doc = ""
output.append(f" {subcommand}: {doc}")
output.sort()
return "\n".join(output)
usage = """
_
| |
_ _|_ | | _ ,_ _ _ _ _ _
|/ | |/ \ |/ / | |/ \_| | / |/ |/ | |/ \_
|__/|_/| |_/|__/ |_/|__/ \_/|_/ | | |_/|__/
/| /|
\| \|
Usage:
etherpump CMD
where CMD could be:
{}
For more information on each command try:
etherpump CMD --help
""".format(
subcommands()
)
def main():
try:
cmd = sys.argv[1]
if cmd.startswith("-"):
args = sys.argv
else:
args = sys.argv[2:]
if len(sys.argv) < 3:
if any(arg in args for arg in ["--help", "-h"]):
print(usage)
sys.exit(0)
elif any(arg in args for arg in ["--version", "-v"]):
print("etherpump {}".format(__VERSION__))
sys.exit(0)
except IndexError:
print(usage)
sys.exit(0)
try:
# http://stackoverflow.com/questions/301134/dynamic-module-import-in-python
cmdmod = __import__(
"etherpump.commands.%s" % cmd, fromlist=["etherdump.commands"]
)
cmdmod.main(args)
except ImportError as e:
print(
"Error performing command '{0}'\n(python said: {1})\n".format(
cmd, e
)
)
print(usage)

8
etherpump/commands/common.py

@ -92,15 +92,15 @@ async def agetjson(session, url):
ret["_url"] = rurl
return ret
except Exception as e:
print('Failed to download {}, saw {}'.format(url, str(e)))
print("Failed to download {}, saw {}".format(url, str(e)))
return
def loadpadinfo(p):
with open(p) as f:
info = json.load(f)
if 'localapiurl' not in info:
info['localapiurl'] = info.get('apiurl')
if "localapiurl" not in info:
info["localapiurl"] = info.get("apiurl")
return info
@ -137,7 +137,7 @@ def unescape(text):
def istty():
return sys.stdout.isatty() and os.environ.get('TERM') != 'dumb'
return sys.stdout.isatty() and os.environ.get("TERM") != "dumb"
def chunks(lst, n):

14
etherpump/commands/creatediffhtml.py

@ -33,20 +33,20 @@ def main(args):
apiurl = info.get("apiurl")
# apiurl = "{0[protocol]}://{0[hostname]}:{0[port]}{0[apiurl]}{0[apiversion]}/".format(info)
data = {}
data['apikey'] = info['apikey']
data['padID'] = args.padid
data['startRev'] = "0"
data["apikey"] = info["apikey"]
data["padID"] = args.padid
data["startRev"] = "0"
if args.rev != None:
data['rev'] = args.rev
requesturl = apiurl + 'createDiffHTML?' + urlencode(data)
data["rev"] = args.rev
requesturl = apiurl + "createDiffHTML?" + urlencode(data)
if args.showurl:
print(requesturl)
else:
try:
results = json.load(urlopen(requesturl))['data']
results = json.load(urlopen(requesturl))["data"]
if args.format == "json":
print(json.dumps(results))
else:
print(results['html'])
print(results["html"])
except HTTPError as e:
pass

10
etherpump/commands/deletepad.py

@ -27,9 +27,9 @@ def main(args):
info = json.load(f)
apiurl = info.get("apiurl")
data = {}
data['apikey'] = info['apikey']
data['padID'] = args.padid
requesturl = apiurl + 'deletePad?' + urlencode(data)
data["apikey"] = info["apikey"]
data["padID"] = args.padid
requesturl = apiurl + "deletePad?" + urlencode(data)
if args.showurl:
print(requesturl)
else:
@ -37,5 +37,5 @@ def main(args):
if args.format == "json":
print(json.dumps(results))
else:
if results['data']:
print(results['data']['text'])
if results["data"]:
print(results["data"]["text"])

26
etherpump/commands/dumpcsv.py

@ -53,10 +53,10 @@ def main(args):
info = json.load(f)
apiurl = info.get("apiurl")
data = {}
data['apikey'] = info['apikey']
requesturl = apiurl + 'listAllPads?' + urlencode(data)
data["apikey"] = info["apikey"]
requesturl = apiurl + "listAllPads?" + urlencode(data)
padids = jsonload(requesturl)['data']['padIDs']
padids = jsonload(requesturl)["data"]["padIDs"]
padids.sort()
numpads = len(padids)
maxmsglen = 0
@ -81,22 +81,22 @@ def main(args):
groupname = ""
padidnogroup = padid
data['padID'] = padid
revisions = jsonload(apiurl + 'getRevisionsCount?' + urlencode(data))[
'data'
]['revisions']
data["padID"] = padid
revisions = jsonload(apiurl + "getRevisionsCount?" + urlencode(data))[
"data"
]["revisions"]
if (revisions == 0) and not args.zerorevs:
continue
lastedited_raw = jsonload(apiurl + 'getLastEdited?' + urlencode(data))[
'data'
]['lastEdited']
lastedited_raw = jsonload(apiurl + "getLastEdited?" + urlencode(data))[
"data"
]["lastEdited"]
lastedited_iso = datetime.fromtimestamp(
int(lastedited_raw) / 1000
).isoformat()
author_ids = jsonload(apiurl + 'listAuthorsOfPad?' + urlencode(data))[
'data'
]['authorIDs']
author_ids = jsonload(apiurl + "listAuthorsOfPad?" + urlencode(data))[
"data"
]["authorIDs"]
author_ids = " ".join(author_ids)
out.writerow(
(padidnogroup, groupname, revisions, lastedited_iso, author_ids)

12
etherpump/commands/gethtml.py

@ -31,16 +31,16 @@ def main(args):
apiurl = info.get("apiurl")
# apiurl = "{0[protocol]}://{0[hostname]}:{0[port]}{0[apiurl]}{0[apiversion]}/".format(info)
data = {}
data['apikey'] = info['apikey']
data['padID'] = args.padid
data["apikey"] = info["apikey"]
data["padID"] = args.padid
if args.rev != None:
data['rev'] = args.rev
requesturl = apiurl + 'getHTML?' + urlencode(data)
data["rev"] = args.rev
requesturl = apiurl + "getHTML?" + urlencode(data)
if args.showurl:
print(requesturl)
else:
results = json.load(urlopen(requesturl))['data']
results = json.load(urlopen(requesturl))["data"]
if args.format == "json":
print(json.dumps(results))
else:
print(results['html'])
print(results["html"])

12
etherpump/commands/gettext.py

@ -31,11 +31,11 @@ def main(args):
apiurl = info.get("apiurl")
# apiurl = "{0[protocol]}://{0[hostname]}:{0[port]}{0[apiurl]}{0[apiversion]}/".format(info)
data = {}
data['apikey'] = info['apikey']
data['padID'] = args.padid # is utf-8 encoded
data["apikey"] = info["apikey"]
data["padID"] = args.padid # is utf-8 encoded
if args.rev != None:
data['rev'] = args.rev
requesturl = apiurl + 'getText?' + urlencode(data)
data["rev"] = args.rev
requesturl = apiurl + "getText?" + urlencode(data)
if args.showurl:
print(requesturl)
else:
@ -45,5 +45,5 @@ def main(args):
if args.format == "json":
print(json.dumps(results))
else:
if results['data']:
sys.stdout.write(results['data']['text'])
if results["data"]:
sys.stdout.write(results["data"]["text"])

26
etherpump/commands/index.py

@ -1,5 +1,4 @@
"""Generate pages from etherpumps using a template"""
import json
import os
import re
@ -14,7 +13,6 @@ from jinja2 import Environment, FileSystemLoader
from etherpump.commands.common import * # noqa
"""
index:
Generate pages from etherpumps using a template.
@ -45,7 +43,7 @@ def splitextlong(x):
if m:
return m.groups()
else:
return x, ''
return x, ""
def base(x):
@ -73,7 +71,7 @@ def url_base(url):
return ret
def datetimeformat(t, format='%Y-%m-%d %H:%M:%S'):
def datetimeformat(t, format="%Y-%m-%d %H:%M:%S"):
if type(t) == str:
dt = dateutil.parser.parse(t)
return dt.strftime(format)
@ -152,7 +150,7 @@ def main(args):
help="include files (experimental)",
)
pg = p.add_argument_group('template variables')
pg = p.add_argument_group("template variables")
pg.add_argument(
"--feedurl",
default="feed.xml",
@ -229,8 +227,8 @@ def main(args):
def metaforpaths(paths):
ret = {}
pid = base(paths[0])
ret['pad'] = ret['padid'] = pid
ret['versions'] = [wrappath(x) for x in paths]
ret["pad"] = ret["padid"] = pid
ret["versions"] = [wrappath(x) for x in paths]
lastedited = None
for p in paths:
mtime = os.stat(p).st_mtime
@ -281,8 +279,8 @@ def main(args):
def has_version(padinfo, path):
return [
x
for x in padinfo['versions']
if 'path' in x and x['path'] == "./" + path
for x in padinfo["versions"]
if "path" in x and x["path"] == "./" + path
]
if args.files:
@ -293,7 +291,7 @@ def main(args):
pads_by_base = {}
for p in args.pads:
# print ("Trying padid", p['padid'], file=sys.stderr)
padbase = os.path.splitext(p['padid'])[0]
padbase = os.path.splitext(p["padid"])[0]
pads_by_base[padbase] = p
padbases = list(pads_by_base.keys())
# SORT THEM LONGEST FIRST TO ensure that LONGEST MATCHES MATCH
@ -309,14 +307,14 @@ def main(args):
if p:
if not has_version(p, x):
print(
"Grouping file {0} with pad {1}".format(x, p['padid']),
"Grouping file {0} with pad {1}".format(x, p["padid"]),
file=sys.stderr,
)
p['versions'].append(wrappath(x))
p["versions"].append(wrappath(x))
else:
print(
"Skipping existing version {0} ({1})...".format(
x, p['padid']
x, p["padid"]
),
file=sys.stderr,
)
@ -378,7 +376,7 @@ def main(args):
with open(versions_by_type["text"]["path"]) as f:
p["text"] = f.read()
except FileNotFoundError:
p['text'] = ''
p["text"] = ""
# ADD IN LINK TO PAD AS "link"
for v in linkversions:
if v in versions_by_type:

6
etherpump/commands/list.py

@ -29,12 +29,12 @@ def main(args):
apiurl = info.get("apiurl")
# apiurl = {0[protocol]}://{0[hostname]}:{0[port]}{0[apiurl]}{0[apiversion]}/".format(info)
data = {}
data['apikey'] = info['apikey']
requesturl = apiurl + 'listAllPads?' + urlencode(data)
data["apikey"] = info["apikey"]
requesturl = apiurl + "listAllPads?" + urlencode(data)
if args.showurl:
print(requesturl)
else:
results = getjson(requesturl)['data']['padIDs']
results = getjson(requesturl)["data"]["padIDs"]
if args.format == "json":
print(json.dumps(results))
else:

8
etherpump/commands/listauthors.py

@ -26,13 +26,13 @@ def main(args):
info = json.load(f)
apiurl = info.get("apiurl")
data = {}
data['apikey'] = info['apikey']
data['padID'] = args.padid
requesturl = apiurl + 'listAuthorsOfPad?' + urlencode(data)
data["apikey"] = info["apikey"]
data["padID"] = args.padid
requesturl = apiurl + "listAuthorsOfPad?" + urlencode(data)
if args.showurl:
print(requesturl)
else:
results = json.load(urlopen(requesturl))['data']['authorIDs']
results = json.load(urlopen(requesturl))["data"]["authorIDs"]
if args.format == "json":
print(json.dumps(results))
else:

36
etherpump/commands/publication.py

@ -1,5 +1,4 @@
"""Generate a single document from etherpumps using a template"""
import json
import os
import re
@ -15,7 +14,6 @@ from jinja2 import Environment, FileSystemLoader
from etherpump.commands.common import * # noqa
"""
publication:
Generate a single document from etherpumps using a template.
@ -50,7 +48,7 @@ def splitextlong(x):
if m:
return m.groups()
else:
return x, ''
return x, ""
def base(x):
@ -78,7 +76,7 @@ def url_base(url):
return ret
def datetimeformat(t, format='%Y-%m-%d %H:%M:%S'):
def datetimeformat(t, format="%Y-%m-%d %H:%M:%S"):
if type(t) == str:
dt = dateutil.parser.parse(t)
return dt.strftime(format)
@ -157,7 +155,7 @@ def main(args):
help="include files (experimental)",
)
pg = p.add_argument_group('template variables')
pg = p.add_argument_group("template variables")
pg.add_argument(
"--feedurl",
default="feed.xml",
@ -235,8 +233,8 @@ def main(args):
def metaforpaths(paths):
ret = {}
pid = base(paths[0])
ret['pad'] = ret['padid'] = pid
ret['versions'] = [wrappath(x) for x in paths]
ret["pad"] = ret["padid"] = pid
ret["versions"] = [wrappath(x) for x in paths]
lastedited = None
for p in paths:
mtime = os.stat(p).st_mtime
@ -287,8 +285,8 @@ def main(args):
def has_version(padinfo, path):
return [
x
for x in padinfo['versions']
if 'path' in x and x['path'] == "./" + path
for x in padinfo["versions"]
if "path" in x and x["path"] == "./" + path
]
if args.files:
@ -299,7 +297,7 @@ def main(args):
pads_by_base = {}
for p in args.pads:
# print ("Trying padid", p['padid'], file=sys.stderr)
padbase = os.path.splitext(p['padid'])[0]
padbase = os.path.splitext(p["padid"])[0]
pads_by_base[padbase] = p
padbases = list(pads_by_base.keys())
# SORT THEM LONGEST FIRST TO ensure that LONGEST MATCHES MATCH
@ -315,14 +313,14 @@ def main(args):
if p:
if not has_version(p, x):
print(
"Grouping file {0} with pad {1}".format(x, p['padid']),
"Grouping file {0} with pad {1}".format(x, p["padid"]),
file=sys.stderr,
)
p['versions'].append(wrappath(x))
p["versions"].append(wrappath(x))
else:
print(
"Skipping existing version {0} ({1})...".format(
x, p['padid']
x, p["padid"]
),
file=sys.stderr,
)
@ -365,11 +363,11 @@ def main(args):
# TODO: make this list non-static, but a variable that can be given from the CLI
customorder = [
'nooo.relearn.preamble',
'nooo.relearn.activating.the.archive',
'nooo.relearn.call.for.proposals',
'nooo.relearn.call.for.proposals-proposal-footnote',
'nooo.relearn.colophon',
"nooo.relearn.preamble",
"nooo.relearn.activating.the.archive",
"nooo.relearn.call.for.proposals",
"nooo.relearn.call.for.proposals-proposal-footnote",
"nooo.relearn.colophon",
]
order = []
for x in customorder:
@ -402,7 +400,7 @@ def main(args):
content = f.read()
# print('content:', content)
# [Relearn] Add pandoc command here?
html = pypandoc.convert_text(content, 'html', format='md')
html = pypandoc.convert_text(content, "html", format="md")
# print('html:', html)
p["text"] = html
# except FileNotFoundError:

60
etherpump/commands/pull.py

@ -1,5 +1,4 @@
"""Check for pads that have changed since last sync (according to .meta.json)"""
import json
import os
import re
@ -18,7 +17,6 @@ import trio
from etherpump.commands.common import * # noqa
from etherpump.commands.html5tidy import html5tidy
"""
pull(meta):
Update meta data files for those that have changed.
@ -173,14 +171,14 @@ async def get_padids(args, info, data, session):
if args.padid:
padids = args.padid
elif args.glob:
url = info['localapiurl'] + 'listAllPads?' + urlencode(data)
url = info["localapiurl"] + "listAllPads?" + urlencode(data)
padids = await agetjson(session, url)
padids = padids['data']['padIDs']
padids = padids["data"]["padIDs"]
padids = [x for x in padids if fnmatch(x, args.glob)]
else:
url = info['localapiurl'] + 'listAllPads?' + urlencode(data)
url = info["localapiurl"] + "listAllPads?" + urlencode(data)
padids = await agetjson(session, url)
padids = padids['data']['padIDs']
padids = padids["data"]["padIDs"]
padids.sort()
return padids
@ -191,7 +189,7 @@ async def handle_pad(args, padid, data, info, session):
if args.no_raw_ext:
raw_ext = ""
data['padID'] = padid
data["padID"] = padid
p = padpath(padid, args.pub, args.group, args.fix_names)
if args.folder:
p = os.path.join(p, padid)
@ -210,15 +208,15 @@ async def handle_pad(args, padid, data, info, session):
contents = await f.read()
meta.update(json.loads(contents))
url = (
info['localapiurl'] + 'getRevisionsCount?' + urlencode(data)
info["localapiurl"] + "getRevisionsCount?" + urlencode(data)
)
response = await agetjson(session, url)
revisions = response['data']['revisions']
if meta['revisions'] == revisions and not args.force:
revisions = response["data"]["revisions"]
if meta["revisions"] == revisions and not args.force:
skip = True
break
meta['padid'] = padid
meta["padid"] = padid
versions = meta["versions"] = []
versions.append(
{"url": padurlbase + quote(padid), "type": "pad", "code": 200,}
@ -226,32 +224,32 @@ async def handle_pad(args, padid, data, info, session):
if revisions is None:
url = (
info['localapiurl'] + 'getRevisionsCount?' + urlencode(data)
info["localapiurl"] + "getRevisionsCount?" + urlencode(data)
)
response = await agetjson(session, url)
meta['revisions'] = response['data']['revisions']
meta["revisions"] = response["data"]["revisions"]
else:
meta['revisions'] = revisions
meta["revisions"] = revisions
if (meta['revisions'] == 0) and (not args.zerorevs):
if (meta["revisions"] == 0) and (not args.zerorevs):
skip = True
break
# todo: load more metadata!
meta['group'], meta['pad'] = splitpadname(padid)
meta['pathbase'] = p
meta["group"], meta["pad"] = splitpadname(padid)
meta["pathbase"] = p
url = info['localapiurl'] + 'getLastEdited?' + urlencode(data)
url = info["localapiurl"] + "getLastEdited?" + urlencode(data)
response = await agetjson(session, url)
meta['lastedited_raw'] = int(response['data']['lastEdited'])
meta["lastedited_raw"] = int(response["data"]["lastEdited"])
meta['lastedited_iso'] = datetime.fromtimestamp(
int(meta['lastedited_raw']) / 1000
meta["lastedited_iso"] = datetime.fromtimestamp(
int(meta["lastedited_raw"]) / 1000
).isoformat()
url = info['localapiurl'] + 'listAuthorsOfPad?' + urlencode(data)
url = info["localapiurl"] + "listAuthorsOfPad?" + urlencode(data)
response = await agetjson(session, url)
meta['author_ids'] = response['data']['authorIDs']
meta["author_ids"] = response["data"]["authorIDs"]
break
except HTTPError as e:
@ -290,13 +288,13 @@ async def handle_pad(args, padid, data, info, session):
pass
if args.all or args.text:
url = info['localapiurl'] + 'getText?' + urlencode(data)
url = info["localapiurl"] + "getText?" + urlencode(data)
text = await agetjson(session, url)
ver = {"type": "text"}
versions.append(ver)
ver["code"] = text["_code"]
if text["_code"] == 200:
text = text['data']['text']
text = text["data"]["text"]
##########################################
## ENFORCE __NOPUBLISH__ MAGIC WORD
@ -387,15 +385,15 @@ async def handle_pad(args, padid, data, info, session):
)
if args.all or args.dhtml:
data['startRev'] = "0"
url = info['localapiurl'] + 'createDiffHTML?' + urlencode(data)
data["startRev"] = "0"
url = info["localapiurl"] + "createDiffHTML?" + urlencode(data)
html = await agetjson(session, url)
ver = {"type": "diffhtml"}
versions.append(ver)
ver["code"] = html["_code"]
if html["_code"] == 200:
try:
html = html['data']['html']
html = html["data"]["html"]
ver["path"] = p + ".diff.html"
ver["url"] = quote(ver["path"])
doc = html5lib.parse(
@ -418,13 +416,13 @@ async def handle_pad(args, padid, data, info, session):
# Process text, html, dhtml, all options
if args.all or args.html:
url = info['localapiurl'] + 'getHTML?' + urlencode(data)
url = info["localapiurl"] + "getHTML?" + urlencode(data)
html = await agetjson(session, url)
ver = {"type": "html"}
versions.append(ver)
ver["code"] = html["_code"]
if html["_code"] == 200:
html = html['data']['html']
html = html["data"]["html"]
ver["path"] = p + ".raw.html"
ver["url"] = quote(ver["path"])
doc = html5lib.parse(
@ -453,7 +451,7 @@ async def handle_pad(args, padid, data, info, session):
async def handle_pads(args):
session = asks.Session(connections=args.connection)
info = loadpadinfo(args.padinfo)
data = {'apikey': info['apikey']}
data = {"apikey": info["apikey"]}
padids = await get_padids(args, info, data, session)
if args.skip:

8
etherpump/commands/revisionscount.py

@ -22,11 +22,11 @@ def main(args):
info = json.load(f)
apiurl = info.get("apiurl")
data = {}
data['apikey'] = info['apikey']
data['padID'] = args.padid
requesturl = apiurl + 'getRevisionsCount?' + urlencode(data)
data["apikey"] = info["apikey"]
data["padID"] = args.padid
requesturl = apiurl + "getRevisionsCount?" + urlencode(data)
if args.showurl:
print(requesturl)
else:
results = json.load(urlopen(requesturl))['data']['revisions']
results = json.load(urlopen(requesturl))["data"]["revisions"]
print(results)

20
etherpump/commands/sethtml.py

@ -51,12 +51,12 @@ def main(args):
# check if it's in fact necessary
requesturl = (
apiurl
+ 'getRevisionsCount?'
+ urlencode({'apikey': info['apikey'], 'padID': args.padid})
+ "getRevisionsCount?"
+ urlencode({"apikey": info["apikey"], "padID": args.padid})
)
results = json.load(urlopen(requesturl))
print(json.dumps(results, indent=2), file=sys.stderr)
if results['code'] != 0:
if results["code"] != 0:
createPad = True
if args.html:
@ -65,15 +65,15 @@ def main(args):
html = sys.stdin.read()
params = {}
params['apikey'] = info['apikey']
params['padID'] = args.padid
params["apikey"] = info["apikey"]
params["padID"] = args.padid
if createPad:
requesturl = apiurl + 'createPad'
requesturl = apiurl + "createPad"
if args.showurl:
print(requesturl)
results = requests.post(
requesturl, params=params, data={'text': ''}
requesturl, params=params, data={"text": ""}
) # json.load(urlopen(requesturl))
results = json.loads(results.text)
print(json.dumps(results, indent=2))
@ -82,14 +82,14 @@ def main(args):
print("limiting", len(text), LIMIT_BYTES, file=sys.stderr)
html = html[:LIMIT_BYTES]
requesturl = apiurl + 'setHTML'
requesturl = apiurl + "setHTML"
if args.showurl:
print(requesturl)
# params['html'] = html
results = requests.post(
requesturl,
params={'apikey': info['apikey']},
data={'apikey': info['apikey'], 'padID': args.padid, 'html': html},
params={"apikey": info["apikey"]},
data={"apikey": info["apikey"], "padID": args.padid, "html": html},
) # json.load(urlopen(requesturl))
results = json.loads(results.text)
print(json.dumps(results, indent=2))

18
etherpump/commands/settext.py

@ -43,15 +43,15 @@ def main(args):
apiurl = info.get("apiurl")
# apiurl = "{0[protocol]}://{0[hostname]}:{0[port]}{0[apiurl]}{0[apiversion]}/".format(info)
data = {}
data['apikey'] = info['apikey']
data['padID'] = args.padid # is utf-8 encoded
data["apikey"] = info["apikey"]
data["padID"] = args.padid # is utf-8 encoded
createPad = False
if args.create:
requesturl = apiurl + 'getRevisionsCount?' + urlencode(data)
requesturl = apiurl + "getRevisionsCount?" + urlencode(data)
results = json.load(urlopen(requesturl))
# print (json.dumps(results, indent=2))
if results['code'] != 0:
if results["code"] != 0:
createPad = True
if args.text:
@ -63,12 +63,12 @@ def main(args):
print("limiting", len(text), LIMIT_BYTES)
text = text[:LIMIT_BYTES]
data['text'] = text
data["text"] = text
if createPad:
requesturl = apiurl + 'createPad'
requesturl = apiurl + "createPad"
else:
requesturl = apiurl + 'setText'
requesturl = apiurl + "setText"
if args.showurl:
print(requesturl)
@ -76,10 +76,10 @@ def main(args):
requesturl, params=data
) # json.load(urlopen(requesturl))
results = json.loads(results.text)
if results['code'] != 0:
if results["code"] != 0:
print(
"setText: ERROR ({0}) on pad {1}: {2}".format(
results['code'], args.padid, results['message']
results["code"], args.padid, results["message"]
)
)
# json.dumps(results, indent=2)

2
etherpump/commands/showmeta.py

@ -1,5 +1,4 @@
"""Extract and output selected fields of metadata"""
import json
import re
import sys
@ -7,7 +6,6 @@ from argparse import ArgumentParser
from .common import * # noqa
"""
Extract and output selected fields of metadata
"""

8
etherpump/commands/status.py

@ -1,12 +1,10 @@
"""Update meta data files for those that have changed"""
import os
from argparse import ArgumentParser
from urllib.parse import urlencode
from .common import * # noqa
"""
status (meta):
Update meta data files for those that have changed.
@ -128,13 +126,13 @@ def main(args):
info = loadpadinfo(args.padinfo)
data = {}
data['apikey'] = info['apikey']
data["apikey"] = info["apikey"]
padsbypath = {}
# listAllPads
padids = getjson(info['apiurl'] + 'listAllPads?' + urlencode(data))['data'][
'padIDs'
padids = getjson(info["apiurl"] + "listAllPads?" + urlencode(data))["data"][
"padIDs"
]
padids.sort()
for padid in padids:

12
padinfo.sample.json

@ -1,12 +0,0 @@
{
"protocol": "http",
"port": 9001,
"hostname": "localhost",
"apiversion": "1.2.9",
"apiurl": "/api/",
"apikey": "8f55f9ede1b3f5d88b3c54eb638225a7bb71c64867786b608abacfdb7d418be1",
"groups": {
"71FpVh4MZBvl8VZ6": {"name": "Transmediale", "id": 43},
"HyYfoX3Q6S5utxs5": {"name": "test", "id": 42 }
}
}

786
poetry.lock

@ -0,0 +1,786 @@
[[package]]
category = "main"
description = "High level compatibility layer for multiple asynchronous event loop implementations"
name = "anyio"
optional = false
python-versions = ">=3.5.3"
version = "1.4.0"
[package.dependencies]
async-generator = "*"
idna = ">=2.8"
sniffio = ">=1.1"
[package.extras]
curio = ["curio (>=0.9)"]
doc = ["sphinx-rtd-theme", "sphinx-autodoc-typehints (>=1.2.0)"]
test = ["coverage (>=4.5)", "hypothesis (>=4.0)", "pytest (>=3.7.2)", "uvloop"]
trio = ["trio (>=0.12)"]
[[package]]
category = "dev"
description = "A small Python module for determining appropriate platform-specific dirs, e.g. a \"user data dir\"."
name = "appdirs"
optional = false
python-versions = "*"
version = "1.4.4"
[[package]]
category = "main"
description = "asks - async http"
name = "asks"
optional = false
python-versions = "*"
version = "2.4.10"
[package.dependencies]
anyio = "<2"
async_generator = "*"
h11 = "*"
[[package]]
category = "main"
description = "Async generators and context managers for Python 3.5+"
name = "async-generator"
optional = false
python-versions = ">=3.5"
version = "1.10"
[[package]]
category = "main"
description = "Classes Without Boilerplate"
name = "attrs"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "20.2.0"
[package.extras]
dev = ["coverage (>=5.0.2)", "hypothesis", "pympler", "pytest (>=4.3.0)", "six", "zope.interface", "sphinx", "sphinx-rtd-theme", "pre-commit"]
docs = ["sphinx", "sphinx-rtd-theme", "zope.interface"]
tests = ["coverage (>=5.0.2)", "hypothesis", "pympler", "pytest (>=4.3.0)", "six", "zope.interface"]
tests_no_zope = ["coverage (>=5.0.2)", "hypothesis", "pympler", "pytest (>=4.3.0)", "six"]
[[package]]
category = "dev"
description = "The uncompromising code formatter."
name = "black"
optional = false
python-versions = ">=3.6"
version = "19.10b0"
[package.dependencies]
appdirs = "*"
attrs = ">=18.1.0"
click = ">=6.5"
pathspec = ">=0.6,<1"
regex = "*"
toml = ">=0.9.4"
typed-ast = ">=1.4.0"
[package.extras]
d = ["aiohttp (>=3.3.2)", "aiohttp-cors"]
[[package]]
category = "main"
description = "Python package for providing Mozilla's CA Bundle."
name = "certifi"
optional = false
python-versions = "*"
version = "2020.6.20"
[[package]]
category = "main"
description = "Foreign Function Interface for Python calling C code."
marker = "os_name == \"nt\" and implementation_name != \"pypy\""
name = "cffi"
optional = false
python-versions = "*"
version = "1.14.3"
[package.dependencies]
pycparser = "*"
[[package]]
category = "main"
description = "Universal encoding detector for Python 2 and 3"
name = "chardet"
optional = false
python-versions = "*"
version = "3.0.4"
[[package]]
category = "dev"
description = "Composable command line interface toolkit"
name = "click"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
version = "7.1.2"
[[package]]
category = "main"
description = "PEP 567 Backport"
marker = "python_version < \"3.7\""
name = "contextvars"
optional = false
python-versions = "*"
version = "2.4"
[package.dependencies]
immutables = ">=0.9"
[[package]]
category = "dev"
description = "the modular source code checker: pep8 pyflakes and co"
name = "flake8"
optional = false
python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,>=2.7"
version = "3.8.4"
[package.dependencies]
mccabe = ">=0.6.0,<0.7.0"
pycodestyle = ">=2.6.0a1,<2.7.0"
pyflakes = ">=2.2.0,<2.3.0"
[package.dependencies.importlib-metadata]
python = "<3.8"
version = "*"
[[package]]
category = "main"
description = "A pure-Python, bring-your-own-I/O implementation of HTTP/1.1"
name = "h11"
optional = false
python-versions = "*"
version = "0.10.0"
[[package]]
category = "main"
description = "HTML parser based on the WHATWG HTML specification"
name = "html5lib"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
version = "1.1"
[package.dependencies]
six = ">=1.9"
webencodings = "*"
[package.extras]
all = ["genshi", "chardet (>=2.2)", "lxml"]
chardet = ["chardet (>=2.2)"]
genshi = ["genshi"]
lxml = ["lxml"]
[[package]]
category = "main"
description = "Internationalized Domain Names in Applications (IDNA)"
name = "idna"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "2.10"