Pumping text from etherpads into publications
A command-line utility that extends the multi writing and publishing functionalities of the etherpad by exporting the pads in multiple formats.
Many pads, many networks
Etherpump is a friendly fork of etherdump, a command line tool written by Michael Murtaugh that converts etherpad pages to files. This fork is made out of curiosities in the tool, a wish to study it and shared sparks of enthusiasm to use it in different situations within Varia.
Etherpump is a stretched version of etherdump. It is a playground in which we would like to add features to the initial tool that diffuse actions of dumping into pumping. So most of all, etherpump is a work-in-progress, exploring potential uses of etherpads to edit, structure and publish various types of content.
Added features are:
- opt-in publishing with the
publicationcommand, that listens to custom magic words such as
See the Change log / notes section for further changes.
Etherpump is a tool that is used from the command line. It pumps all pads of one etherpad installation to a folder, saving them as different text files, such as plain text and HTML. It also creates an index file, that allows one to easily navigate through the list of pads. Etherpump follows a document-driven idea of publishing, which means that it converts pads as database entries into pads as files. This seems to be a redundant act of copying, but is actually an important in-between step that allows for many different publishing projects and experiments.
We started to get to know etherpump through various editions of Relearn and/or the worksessions organized by Constant. Collaborative writing on an etherpad has been an important ingredient for these situations. The habit of using pads branched into the day-to-day practice of Varia, where we use etherpads for all sorts of things, ranging from organising remote-meetings with 10+ people, to writing and designing PDF documents collaboratively.
After installing etherpump on the Varia server, we collectively decided to not want to publish pads by default. Discussions in the group around the use of etherpads, privacy and ideas of what publishing means, led to a need to have etherpump only start the indexing work after it recognizes a
__PUBLISH__ marker on a pad. We decided to work on a
__PUBLISH__ vs. __NOPUBLISH__ branch of etherdump, which we now fork into etherpump.
Change log / notes
Use the more friendly packaging tool Poetry for publishing.
Added experimental trio and
asks support for the
command which enables pads to be processed concurrently. The default
--connection option is set to 20 which may overpower the target server. If in
doubt, set this to a lower number (like 5). This functionality is experimental,
be cautious and please report bugs!
Removed fancy progress bars for pulling because concurrent processing makes that hard to track. For now, we simply output whichever padid we're finished with.
etherpump --help handling to make it easier for new users.
Added a fancy progress bar with
tqdm for long running
etherpump pull --all calls
Started with the experimental library API.
Forking etherpump into etherpump.
Migrating the source code to Python 3.
Integrate PyPi publishing with setuptools.
May - September 2019
etherpump is used to produce the Ruminating Relearn section of the Network Of One's Own 2 (NOOO2) publication.
A new command is added to make a web publication, based on the custom magic word
Multiple conversations around etherpump emerged during Relearn Curved in Varia, Rotterdam.
Including the idea of executable pads (etherhooks), custom magic words, a federated snippet protocol (etherstekje) and more.
Installation of etherpump on the Varia server.
__PUBLISH__ vs. __NOPUBLISH__ was added to the etherpump repository by decentral1se.
Originally designed for use at: Constant.
More notes can be found in the git repository of etherdump.
$ pip install etherpump
Etherpump only supports Python 3.
$ mkdir mydump $ cd myddump $ etherpump init
The program then interactively asks some questions:
Please type the URL of the etherpad: https://pad.vvvvvvaria.org/
The APIKEY is the contents of the file APIKEY.txt in the etherpad folder.
Please paste the APIKEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
The settings are placed in a file called
.etherpump/settings.json and are used (by default) by future commands.
Library API Example
Etherpump can be used as a library.
All commands can be imported and run programmatically.
>>> from etherpump.api import pull >>> pull(['--all', '--publish-opt-in', '--publish', '__PUB_CLUB__'])
To see all available subcommands, run:
$ etherpump --help
For help on each individual subcommand, run:
$ etherpump revisionscount --help
Please use "semver" conventions for versions.
Here are the steps to follow (e.g. for a
- Change the version number in the
- Change the version number in the
git add . && git commit -m "Publish new 0.1.3 version" && git tag 0.1.3 && git push --tags
poetry publish --build
Tools to help things stay tidy over time.
Please see the following links for further reading:
Keeping track of Etherpad-lite
GNU AFFERO GENERAL PUBLIC LICENSE, Version 3.