pacserve

2013-09-18 14:46 UTC
  • Xyne

Metadata

Description:

Easily share Pacman packages between computers. A replacement for PkgD.

Latest Version:

2013.9

Architecture:

  • any

Dependencies:

Arch Repositories:

  • [xyne-any]
  • [xyne-i686]
  • [xyne-x86_64]

AUR ID:

48294

Arch Forum ID:

117094

Tags:

About

Pacserve is a simple server intended for use on LANs with multiple Arch Linux systems. It makes it easy to share packages among local computers to avoid redundant downloads from mirrors and thus save bandwidth. It is a replacement for the deprecated PkgD project.

With the Python 3 rewrite, Pacserve is now just a thin subclass (read: wrapper) of Quickserve, so please read the Quickserve page for usage, screenshot and whatever else is there. All Quickserve options should work with Pacserve.

Backend

As of pacserve-2013.5, the backend has been completely rewritten and moved to python3-threaded_servers.

How It Works

The following flowchart gives an overview of how each pacserver handles a request from Pacman:

svg image

Packages are thus retrieved from the local network if possible and no bandwidth is wasted on redundant downloads from external mirrors.

Non-package requests are passed directly through to the first mirror that can fulfill the request.

Other Pacservers are queried using custom searches and redundant and recursive queries are avoided.

Simple Setup Example

svg image

Let's say that you have a desktop computer and two laptops, each running Arch Linux. All three connect to your router. To share packages between them, do the following on each system:

  1. If you have an iptables-based firewall, enabled the pacserve-ports service:

    systemctl enable pacserve-ports.service`

  2. Start the pacserve service:

    systemctl start pacserve.service

  3. Use pacsrv exactly as you would use pacman, e.g.:

    pacsrv -Syu

After that it should "just work". To start pacserve automatically when the system reboots, enable the pacserve.service with systemctl enable pacserve.service. Each system runs its own pacserver and pacman only queries the pacserver on the same system. Pacserve handles all communication between the different systems. Other pacservers will be detected automatically using multicast if they join the network.

If you need anything more advanced than that then you should be able to figure it out using pacserve --help and by creating your own service file. If not, send me an email and I will try to help.

Local Master Repositories

If you have one master computer that should set the pace of upgrades for all other servers of the same architecture, then you can use a filelist to serve database files along with packages. The following filelist will include the local databases in the servers /pkgs/ directory:

{
  "pkg" : ["/var/lib/pacman/sync"]
}

All servers that point to this Pacserve server will download the database files from the local server. The local server should not use the same Pacserve server as it will only find it's own databases and may even overwrite them if it ignores the Last-Modified header (which is unlikely, but some people use crazy configurations). When the local server updates from a remote server, all the local clients will synchronize with it on the next upgrade.

Of course, if the client queries packages that are not on the local server then it may be redirected elsewhere, but this can also be controlled by passing Pacserve a custom local mirrorlist (or no mirrors at all) via the --pacman-conf argument.

Of course, if you just want to set up custom local repositories then Quickserve is likely a better tools.

Pacman Configuration File

If you do not want to use the pacsrv wrapper script (see below) then you will need to manually add Pacserve server lines to your pacman.conf file. See the output of pacman.conf-insert_pacserve to see how it's done. Each repo section will require a line like this:

Include = /etc/pacman.d/pacserve

For the official mirrors, it can be added to the top of the Pacman mirrorlist file. For others, it should follow immediately after the repo section header, e.g.

[xyne-any]
Include = /etc/pacman.d/pacserve

This is what pacman.conf-insert_pacserve does automatically, but the generated file is cached and used by pacsrv instead of modifying the system pacman.conf file. This way you can use pacsrv when the server is running and pacman when it is not.

XferCommand

At the time of writing, the example XferCommands included in pacman.conf do not properly quote the path (%o) and URL (%u) placeholders. This will prevent the commands from working with pacserve. The following will work:

XferCommand = /usr/bin/curl -C - -f '%u' > '%o'
XferCommand = /usr/bin/wget --passive-ftp -c -O '%o' '%u'
XferCommand = /usr/bin/axel -n 8 -a -v -o '%o' '%u'

Of course, I strongly recommend using powerpill instead of setting XferCommands as it has internal support for pacserve. It will download all available packages via pacserve, then use parallel and segmented downloads to retreive the rest.

Opening Firewall Ports

Pacserve normally requires 2 ports:

  • 15678 - for incoming HTTP(S) connections
  • 15679 - for incoming multicast announcements

If you are running systemd and use iptables.service, you can enable pacserve-ports.service to open and close ports automatically when the pacserve.service starts and stops, respectively. You can also use pacserve-ports to manually open and close ports (see below). The rules can be configured in /etc/conf.d/pacserve.

Configuration

  • /etc/conf.d/pacserve - generation configuration file for services and scripts
  • /etc/pacman.d/pacserve - Pacman configuration file server parameter

Convenience Scripts

Pacserve comes with two convenience scripts: pacsrv and pacman.conf-insert_pacserve

pacsrv

pacsrv is a pacman wrapper. It can detect if Pacserve is running and use it automatically. It creates a modified copy of the Pacman configuration file with a Pacserve entry for each repo. The modified copy is kept in sync with the specified Pacman configuration file.

pacman.conf-insert_pacserve

This script is used by pacsrv to insert

Include = /etc/pacman.d/pacserve

into each repo section in a Pacman configuration file. It will read the inputfile (/etc/pacman.conf by default) and print the modified version to STDOUT.

pacserve-ports

This will parse the rules in /etc/conf.d/pacserve and open or close them. It is used by pacserve-ports.service.

Ideas & TODO

  • maybe change the way database requests are handled
  • maybe enable more advanced options for connecting to other Pacservers, such as using client certificates

Help Message

$ pacserve --help

usage: Pacserve.py [-h] [--pacman-conf <filepath>] [--trust-pacserve-peers]
                   [--root <directory path>] [-f <filepath>]
                   [--filter <ix><regex>] [--filterlist <filepath>]
                   [--show-hidden] [--upload <filepath>] [--allow-overwrite]
                   [--motd <filepath>] [--index <filename>]
                   [--peer <scheme>://<host>:<port>/] [--list-remote]
                   [-a <interface|address>] [-p <port>] [--ipv6]
                   [--auth <string> <string>] [--authfile <filepath>] [--ssl]
                   [--certfile <filepath>] [--keyfile <filepath>] [--req-cert]
                   [--ca-certs <filepath>] [--multicast]
                   [--multicast-address <interface|address>]
                   [--multicast-port <port>] [--multicast-group <group>]
                   [--multicast-interval <seconds>]
                   [--multicast-interface <interface|address>]
                   [<filepath> [<filepath> ...]]

Pacserve.py - share Pacman packages over your LAN and beyond

positional arguments:
  <filepath>            Additional files and directories to share. These will
                        appear with the same name in server root. Use the
                        filelist option for more advanced features.

optional arguments:
  -h, --help            show this help message and exit

Pacserve Options:
  --pacman-conf <filepath>
                        The Pacman configuration file to use. Default:
                        /etc/pacman.conf
  --trust-pacserve-peers
                        Allow the server to redirect database, signature and
                        other non-package request to its peers instead of
                        immediately redirecting to a mirror. This can be
                        useful for some setups but you should only use it if
                        you trust the peers or know exactly what you are
                        doing.

File Download Options:
  --root <directory path>
                        If given then the directory will be treated as the
                        root of the server and all other paths will be
                        ignored. This is useful for testing static websites.
                        Similar and more complicated effects can be achieved
                        using a JSON filelist.
  -f <filepath>, --filelist <filepath>
                        A file to specify what to share on the server. If it
                        is a flat plaintext file then each line will be
                        treated as though it had been passed on the command
                        line. If it is a JSON file then it should be a map of
                        server paths to either single files or lists of
                        directories. The contents of each directory in the
                        list will appear as a single directory on the server.
  --filter <ix><regex>  Regular expressions to filter paths that appear on the
                        server. These will be applied in order when
                        determining which files to share.
  --filterlist <filepath>
                        A file consisting of filter expressions on each line.
                        The file will be reloaded if it is modified.
  --show-hidden         Share hidden files and directories.

File Upload Options:
  --upload <filepath>   Enable uploads and save uploaded files in given
                        directory.
  --allow-overwrite     Allow uploaded files to overwrite existing files in
                        upload directory.

Content Options:
  --motd <filepath>     The MOTD message to display on the server. The file
                        will be reloaded if it is updated.
  --index <filename>    The name of the index page to display (if present)
                        when a directory is requested.

MulticastQuickserve Options:
  --peer <scheme>://<host>:<port>/
                        Static peers. Pass the option multiple times if
                        necessary. Example: "http://10.0.0.2:8000/"
  --list-remote         Include remote files in directory listings.

Server Address and Port:
  Configure the server's listening address and port.

  -a <interface|address>, --address <interface|address>
                        Bind the server to this address. By default the server
                        will listen on all interfaces.
  -p <port>, --port <port>
                        Set the server port (default: 15678)
  --ipv6                Use IPv6.

HTTP Authentication:
  HTTP digest authentication via a username and password.

  --auth <string> <string>
                        HTTP digest username and password. Multiple pairs may
                        be passed.
  --authfile <filepath>
                        The path to a file containing alternating lines of
                        usernames and passwords.

SSL (HTTPS):
  Options for wrapping sockets in SSL for encrypted connections. Simply
  enabling SSL does not guarantee a secure connection and it is the user's
  responsibility to check that the implementation is correct and secure and
  that the server is properly configured. You can find information about
  generating self-signed certificates in the OpenSSL FAQ:
  http://www.openssl.org/support/faq.html

  --ssl                 Enable SSL (HTTPS).
  --certfile <filepath>
                        The path to the server's certificate.
  --keyfile <filepath>  The path to the server's key.
  --req-cert            Require a certificate from the client.
  --ca-certs <filepath>
                        Set the path to a file containing concatenated CA
                        certificates for verifying the client certificate.
                        This defaults to the server's own certificate.

Multicast Options:
  Options that affect the behavior of the multicast (sub)server system.

  --multicast           Use multicasting to announce presence and detect other
                        servers.
  --multicast-address <interface|address>
                        The address to which to bind the multicast server
                        socket. Default: 0.0.0.0.
  --multicast-port <port>
                        The multicast port. Default: 15679.
  --multicast-group <group>
                        The multicast group. Default: 224.3.45.67.
  --multicast-interval <seconds>
                        The multicast announcement interval. Default: 300.
  --multicast-interface <interface|address>
                        The interface or address through which to announce
                        presence with multicast packets. If not given, all
                        interfaces are used.

CHANGELOG

2013-05-10

  • completely rewritten in Python 3
  • better handling of inter-server notification via HTTP POST and UDP multicasting
  • some option changes
  • now serves HTML directory listings

2013-04-14

  • removed comments from service environment file to work around systemd bug

2013-04-13

  • split configuration file and moved to /etc/pacserve/
  • added RestartSet to pacserve.service file

2012-11-27

  • moved /etc/conf.d/pacserved to /etc/conf.d/pacserve
  • modified /etc/conf.d/pacserve (now manages global options for scripts)
  • modified service file to use configuration file
  • added /etc/pacman.d/pacserve as central pacman.conf include file
  • added --multicast-interval option for periodically announcing presence

2012-11-25

  • added pacserve-ports default port opener script
  • added pacserve-ports.service

2012-07-17

  • added pacsrv script
  • added pacman.conf-insert_pacserve script

2012-07-16

  • restructured code to correct order of forking and threading, which previously prevented opening of the log file

2012-06-04

  • fixed bug in filename percent encoding
  • added daemon (/etc/rc.d/pacserved) in part inspired by a contributed daemon from Willow Walthall
  • added daemon-related options (--su, --daemon, --pid-file, --log)

2012-03-26

  • included the service file submitted by Jameson

2011-04-19

  • added multicast support, inspired by toofishes' multipkg
  • changed default port to 15678 instead of inheriting 8080 from XyneHTTPServer
Contact
echo xyne.archlinux.ca | sed 's/\./@/'
Feeds
Blog News
Validation
XHTML 1.0 Strict CSS level 3 Atom 1.0