Python Packaging 一些阅读笔记

Python Packaging

  • Python 包管理工具怎样管理一个包的元信息？要考虑哪些问题（versioning, naming…）？
  • wheel vs eggs? (50%)
  • wheel vs sdist? (50%)
  • 怎样优雅的发布 .proto/.thrift 等 RPC 接口定义文件？

相关 PEP

PEP 376 Notes (eggs-info go die)

这个 PEP 目标是提供一个管理系统上安装的包发行版 (distribution) 的标准基础设施， 让其他安装、卸载包的工具是可以相互协同工作的。

为了实现这个目标，该 PEP 提出一种新的格式来描述一个安装在系统上的分发。 该 PEP 也描述了标准库的参考实现。

这个 PEP 结合了 PEP 345，可以取代 PEP 262。

基础原理

目前 Python 安装包发行版的方式存在两个问题：

  1. 安装方式太多，标准不一，各种工具不能协同工作
  2. 没有 API 可以获取已经安装的包的信息
目前包发行版的安装方式

当前，在安装一个包的发行版时，其每个元素（文件）可以被放到不同的目录下。

例如， Distutils 会把 Python 代码放到 purelib 目录下， 在 unix-like 和 osx 中，即 lib\python2.6\site-packages 目录；对于 Windows 来说，就是 Lib/site-packages 目录。

另外， Distutils 的安装子命令 install_egg_info 会给每个包发行版 加一个 .egg-info 文件加入到 purelib 目录。

举个例子：对于 docutils 这个发行版，它包含以下内容

  • docutils: The docutils package
  • roman.py: An extra module used by docutils
  • docutils-0.5-py26.egg-info: A file containing the distribution metadata as described in PEP 314 . This file corresponds to the file called PKG-INFO, built by the sdist command.

一些可执行脚本，比如 rst2html.py ，被放到 Python 的 bin 目录。

另外一个项目， setuptools 它有两种其它的格式来安装包发行版，叫做 EggFormats。 包含以下内容：

  • 一个自包含的 .egg 目录。它的子目录 EGG-INFO 中的 PKG-INFO 文件 包含所有的发行文件和发行元信息。setuptools 会在 .egg 目录创建 其它文件作为元信息的补充。
  • 一个安装在 site-packages 目录的 .egg-info 目录，这里面包含了同样 的 EGG-INFO 目录。

setuptools 还会在 easy-install.pth 文件中为该包发行版加一个引用。

卸载包发行版

Distutils 没有提供卸载的命令。要想卸载一个包，需要手动删除文件，编辑 .pth 文件。 手动卸载问题很多，省略之。

但这些工具有个共同特性：它们安装一个包的发行版时，它们都可以记录这些文件， 以后删除的时候，就知道需要删除哪些东西了。

此外，Pip 项目后来增加了一个卸载功能。它可以记录所有安装的文件。

这个 PEP 提出的方案
  • 一个新的 .dist-info 目录结构，灵感来自于 EggFormats 标准
  • pkgutils 包含新的 API 可以查询已经安装的包发行版的信息
  • 在 distutils 中添加了 uninstall 函数和 uninstall 脚本
One .dist-info directory per installed distribution

实现细节

PEP 427 Notes

setup.py 各个命令

怎样实现 .proto/.thrift 的优雅发布？

FQA

打包时需要需要把 pyc 带入包内？

pyc 跨平台，不跨 python 版本 eggs 是会带 pyc 的

Wheel archives do not include .pyc files. Therefore, when the distribution only contains Python files (i.e. no compiled extensions), and is compatible with Python 2 and 3, it’s possible for a wheel to be “universal”, similar to an sdist.

Why not eggs?

  • Wheel has an official PEP. Egg did not.
  • Wheel is a distribution format, i.e a packaging format. [1] Egg was both a distribution format and a runtime installation format (if left zipped), and was designed to be importable.
  • Wheel archives do not include .pyc files. Therefore, when the distribution only contains Python files (i.e. no compiled extensions), and is compatible with Python 2 and 3, it’s possible for a wheel to be “universal”, similar to an sdist.
  • Wheel uses PEP376-compliant .dist-info directories. Egg used .egg-info.
  • Wheel has a richer file naming convention. A single wheel archive can indicate its compatibility with a number of Python language versions and implementations, ABIs, and system architectures.
  • Wheel is versioned. Every wheel file contains the version of the wheel specification and the implementation that packaged it.
  • Wheel is internally organized by sysconfig path type, therefore making it easier to convert to other formats.

Wheel vs sdist

see pep 427

Python needs a package format that is easier to install than sdist. Python’s sdist packages are defined by and require the distutils and setuptools build systems, running arbitrary code to build-and-install, and re-compile, code just so it can be installed into a new virtualenv. This system of conflating build-install is slow, hard to maintain, and hinders innovation in both build systems and installers.

setup_requires vs tests_require vs install_requires

setup_requires 运行 setup.py 脚本的时候会下载这些依赖，但不会安装

A string or list of strings specifying what other distributions need to be present in order for the setup script to run. setuptools will attempt to obtain these (even going so far as to download them using EasyInstall) before processing the rest of the setup script or commands. This argument is needed if you are using distutils extensions as part of your build process; for example, extensions that process setup() arguments and turn them into EGG-INFO metadata files.

(Note: projects listed in setup_requires will NOT be automatically installed on the system where the setup script is being run. They are simply downloaded to the ./.eggs directory if they’re not locally available already. If you want them to be installed, as well as being available when the setup script is run, you should add them to install_requires and setup_requires.)

tests_require: setup.py test 跑测试的时候会下载

If your project’s tests need one or more additional packages besides those needed to install it, you can use this option to specify them. It should be a string or list of strings specifying what other distributions need to be present for the package’s tests to run. When you run the test command, setuptools will attempt to obtain these (even going so far as to download them using EasyInstall). Note that these required projects will not be installed on the system where the tests are run, but only downloaded to the project’s setup directory if they’re not already installed locally.

install_requires

Updated:

Comments