Fix failing CI

[NicholasBHubbard]

This PR fixes the failing CI, solving issue #2121.

The reason the CI was failing is that Python pyproject.toml support was recently added to fpm, but the version of the Python wheel package that the Ubuntu 22.04 GitHub runner packaged was an older version that didn't have necessary pyproject.toml support. This could have been solved by configuring the CI to use a Python venv then install the newer version of wheel into the venv. While I was already trying to fix the CI though, I thought we may as well update the runner to use the newer Ubuntu 24.04, which happens to also packages a newer version of wheel that has pyproject.toml support.

Updating to Ubuntu 24.04 caused a few more breakages though.

One breakage we got was that one of the deb tests creates a fake package with lsb-base as a dependency. The newer lintian version for Ubuntu 24.04 complains that this lsb-base package is obsolete. I fixed this by changing the dependency to debconf as this package is unlikely to go obsolete any time soon.

The other breakage was that Ubuntu 24.04 uses the PEP 668 feature of marking the Python system packages as EXTERNALLY-MANAGED, and one of our test modules (click) was already installed into the system Python on the GitHub runner. When one of our tests went to install click again with pip we got the following failure:

$ python -m pip install --no-deps --root /tmp/package-python-build /tmp/package-python-staging/click-8.3.0-py3-none-any.whl
Processing /tmp/package-python-staging/click-8.3.0-py3-none-any.whl
Installing collected packages: click
  Attempting uninstall: click
    Found existing installation: click 8.1.6
    Uninstalling click-8.1.6:
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'termui.py'
Consider using the `--user` option or check the permissions.

There were two solution I found for this. The first was to just use a different test package other than click that wasn't already installed on the system. The other option (that I went with) was to setup a venv in the CI runner before running the test suite. I thought that using a venv was a more normal and correct solution that should continue to work moving forward regardless of changes to the CI runners Python environment. One side-effect of using a venv though is that the packaging package was not installed into the venv (tangentially related to issue #2118). This wasn't a problem before because the Ubuntu 22.04 runner happened to already have this installed. However, packaging isn't fpm's only Python package dependency, fpm also has a Python script lib/fpm/package/pyfpm/get_metadata.py that depends on simplejson unless we already have the core json package available. Further, fpm has a test Python script spec/fixtures/python/setup.py that depends on setuptools. This means that we have a test-specific Python dependency. There are two ways I thought of to solve this. The first is in the CI configuration after setting up the venv to just explicitly install these 3 packages: $ pip install setuptools packaging simplejson. The other solution that I went with is more complicated, and it is to add a requirements.txt file to the root of the fpm repository, and a requirements.txt to the spec/ directory for the test dependency. This makes it easy for fpm users to get their Python dependencies, while also making it easy to add new dependencies in the future. I do understand that this solution is more complicated and out of scope of simply fixing the CI, so I am very open to feedback. I added a short mention of this to the docs in the optional dependencies section of the installation manual.

Let me know how everything looks and if you would like anything to be changed. Thanks!

[wbraswell]

@jordansissel
I have reviewed and approved this PR, thanks in advance! :-)

[jordansissel]

I appreciate your effort to battle the black box that is GitHub Runners!

Ubuntu 22.04

for Python wheel: is this finding a real problem for fpm and not just a problem on CI? Because as I read it, fpm can’t create Python packages when run on Ubuntu 22.04 with the default wheel version? This feels like a bug correctly identified - that this older wheel isn’t supported (this wasn’t an intentional choice)

I fixed this by changing the dependency to debconf

+1

and one of our test modules (click) was already installed into the system Python

Feels like a bug. Fpm should be able to package click even if click is already installed. I’ll read more about your venv idea soon.

fpm also has a Python script lib/fpm/package/pyfpm/get_metadata.py that depends on simplejson

these files I think aren’t used anymore since a recent change moved fpm from using setup.py to only use pip. It’s worth another review, but I believe they’re fine to delete (the tests and get_metadata.py)

[NicholasBHubbard]

Thanks for the response @jordansissel!

I'll go through and respond to all of your comments.

for Python wheel: is this finding a real problem for fpm and not just a problem on CI? Because as I read it, fpm can’t create Python packages when run on Ubuntu 22.04 with the default wheel version? This feels like a bug correctly identified - that this older wheel isn’t supported (this wasn’t an intentional choice)

I would say that the fact that we need a newer version of wheel is not a bug in fpm, because the new feature is for pyproject.toml Python projects, and this is a new feature in Python. This feature is from PEP 621 which was first created recently on June 22 2020. I would say that fpm users expecting support for a newer Python feature should be expected to be using a newer and up-to-date platform.

and one of our test modules (click) was already installed into the system Python

Feels like a bug. Fpm should be able to package click even if click is already installed. I’ll read more about your venv idea soon.

I agree with you, this does feel like it's an fpm bug. It seems to be the case that the fpm Python code should run in a temporary virtual environment. We would also need to detect if the user is already running in a virtual environment to prevent nested virtual environments. Pip version pip 23.0 (released on 01/30/2023, between our two Ubuntu releases) implemented PEP 668. The following Dockerfile can be used to reproduce the bug, which gives us an error message from pip specifically telling us we should use a virtual environment (importantly we do not get any error if we switch the base image to ubuntu:22.04):

FROM ubuntu:24.04

RUN apt-get -y update
RUN apt-get -y install python3-pip

RUN useradd -m -s /bin/bash foouser
USER foouser

RUN pip install click

CMD ["/bin/bash"]

$ docker build -t testing .
...
> [5/5] RUN pip install click:
0.862 error: externally-managed-environment
0.862
0.862 × This environment is externally managed
0.862 ╰─> To install Python packages system-wide, try apt install
0.862     python3-xyz, where xyz is the package you are trying to
0.862     install.
0.862
0.862     If you wish to install a non-Debian-packaged Python package,
0.862     create a virtual environment using python3 -m venv path/to/venv.
0.862     Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
0.862     sure you have python3-full installed.
0.862
0.862     If you wish to install a non-Debian packaged Python application,
0.862     it may be easiest to use pipx install xyz, which will manage a
0.862     virtual environment for you. Make sure you have pipx installed.
0.862
0.862     See /usr/share/doc/python3.12/README.venv for more information.
0.862
0.862 note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
0.862 hint: See PEP 668 for the detailed specification.

If we updated the fpm Python code to use a virtual environment, then we would not need to setup the virtual environment in the CI.

So it seems that our two options are to require the user to use a virtual environment when using fpm to package Python projects, or we upgrade fpm to automatically utilize a virtual environment when building Python packages.

fpm also has a Python script lib/fpm/package/pyfpm/get_metadata.py that depends on simplejson

these files I think aren’t used anymore since a recent change moved fpm from using setup.py to only use pip. It’s worth another review, but I believe they’re fine to delete (the tests and get_metadata.py)

We double checked and confirmed that lib/fpm/package/pyfpm/get_metadata.py is no longer being used:

$ fgrep -I -r get_metadata.py; echo $?
1

commit 8e27610 removes this file and the dependency on simplejson.

We also realized that fpm does not actually have a dependency on setuptools since it is a dependency of pip, which is what fpm actually depends on.

So to recap, our main question for you right now is as follows: do we want fpm itself to have the ability to detect and create Python virtual environments, or is setting up a Python virtual environment the sole responsibility of each fpm end user?

[wbraswell]

@jordansissel
As usual, I worked directly with @NicholasBHubbard to ensure this latest commit and message are on point, thanks in advance for helping get this merged!

[jordansissel]

So to recap, our main question for you right now is as follows: do we want fpm itself to have the ability to detect and create Python virtual environments, or is setting up a Python virtual environment the sole responsibility of each fpm end user?

In such cases, fpm should create (and destroy) the virtual environment because fpm would be doing this to work around some kind of problem that, ideally, the user is unaware of.

However, see below:

If we updated the fpm Python code to use a virtual environment, then we would not need to setup the virtual environment in the CI.

I don't think we need this, and worry a virtual environment would cause different problems once the python package is pulled out of the virtual env... it's worth testing, though.

Without a virtual env -- I did a little bit of digging. I think fpm should be using the --target flag, not --root {staging_path} -- it'll have to compute the correct install path in order to use --target correctly, though I can't find my notes on that at this time.

# Install an already installed-elsewhere package, fails:
% python3 -m pip install --no-deps --root /tmp --prefix /tmp/z/usr vine
Requirement already satisfied: vine in /home/jls/.local/lib/python3.13/site-packages (5.1.0)

# Using --target instead

If we updated the fpm Python code to use a virtual environment, then we would not need to setup the virtual environment in the CI.

% python3 -m pip install --no-deps --target /tmp/z/python vine
Collecting vine
  Using cached vine-5.1.0-py3-none-any.whl.metadata (2.7 kB)
Using cached vine-5.1.0-py3-none-any.whl (9.6 kB)
Installing collected packages: vine
Successfully installed vine-5.1.0

If we have to setup a virutal env, I accept that, but it may bring new challenges, and I'm not sure what side effects that may have (rewriting #! shebang lines? files placed where they aren't expected? A venv isn't relocatable, etc)

[jordansissel]

I would say that the fact that we need a newer version of wheel is not a bug in fpm

because the new feature is for pyproject.toml Python projects, and this is a new feature in Python.

I'd like to understand more about this. It was not intended that older versions do not work.

If I try myself on Ubuntu 22.04 in docker, python3 and wheel 0.37.1 (via dpkg/apt), packaging django fails:

Traceback (most recent call last): {:level=>:warn, :file=>"cabin/mixins/pipe.rb", :line=>"47", :method=>"block in pipe"}
  File "/var/lib/gems/3.0.0/gems/fpm-1.17.0/lib/fpm/package/pyfpm/parse_requires.py", line 24, in <module> {:level=>:warn, :file=>"cabin/mixins/pipe.rb", :line=>"47", :method=>"block in pipe"}
    print(json.dumps(list(evaluate_requirements(sys.stdin)))) {:level=>:warn, :file=>"cabin/mixins/pipe.rb", :line=>"47", :method=>"block in pipe"}
  File "/var/lib/gems/3.0.0/gems/fpm-1.17.0/lib/fpm/package/pyfpm/parse_requires.py", line 17, in evaluate_requirements {:level=>:warn, :file=>"cabin/mixins/pipe.rb", :line=>"47", :method=>"block in pipe"}
    if req.marker is None or req.marker.evaluate(): {:level=>:warn, :file=>"cabin/mixins/pipe.rb", :line=>"47", :method=>"block in pipe"}
  File "/usr/lib/python3/dist-packages/packaging/markers.py", line 304, in evaluate {:level=>:warn, :file=>"cabin/mixins/pipe.rb", :line=>"47", :method=>"block in pipe"}
    return _evaluate_markers(self._markers, current_environment) {:level=>:warn, :file=>"cabin/mixins/pipe.rb", :line=>"47", :method=>"block in pipe"}
  File "/usr/lib/python3/dist-packages/packaging/markers.py", line 234, in _evaluate_markers {:level=>:warn, :file=>"cabin/mixins/pipe.rb", :line=>"47", :method=>"block in pipe"}
    lhs_value = _get_env(environment, lhs.value) {:level=>:warn, :file=>"cabin/mixins/pipe.rb", :line=>"47", :method=>"block in pipe"}
  File "/usr/lib/python3/dist-packages/packaging/markers.py", line 215, in _get_env {:level=>:warn, :file=>"cabin/mixins/pipe.rb", :line=>"47", :method=>"block in pipe"}
    raise UndefinedEnvironmentName( {:level=>:warn, :file=>"cabin/mixins/pipe.rb", :line=>"47", :method=>"block in pipe"}
packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment. {:level=>:warn, :file=>"cabin/mixins/pipe.rb", :line=>"47", :method=>"block in pipe"}

This error comes from fpm's python script that tries to parse the package's metadata. However, I see that installing django with pip directly works fine:

root@0031030bac38:/# python3 -m pip install --target /tmp/django --no-deps django
Collecting django
  Using cached django-5.2.8-py3-none-any.whl (8.3 MB)
Installing collected packages: django
Successfully installed django-5.2.8

If this error is something you were addressing by upgrading to Ubuntu 24.04, then this further confirms my belief that something about this is a bug in fpm or needs further investigation.

[jordansissel]

My time is limited lately, but I'm attending to these as best I can. Most days I get maybe 30 minutes, usually less, to attend to any computer-related tasks, including open source issue investigations.

[NicholasBHubbard]

My time is limited lately, but I'm attending to these as best I can. Most days I get maybe 30 minutes, usually less, to attend to any computer-related tasks, including open source issue investigations.

Thanks! We really appreciate the help!

[NicholasBHubbard]

Thanks so much for the helpful response @jordansissel!

I don't think we need this, and worry a virtual environment would cause different problems once the python package is pulled out of the virtual env... it's worth testing, though.

Without a virtual env -- I did a little bit of digging. I think fpm should be using the --target flag, not --root {staging_path} -- it'll have to compute the correct install path in order to use --target correctly, though I can't find my notes on that at this time.

I did some more digging, and I found that you are right that we don't need to use a virtual environment to solve this particular problem. I tried to change the pip command that is run in python.rb to use --target instead of --root + --prefix, however I ran into the problem you mentioned about computing the correct install path. I tried using the python sysconfig library which can find the python platlib and purelib which are where we install platform-specific and non-platform-specific python code respectively. However, determining if if the python package we are installing is platform-specific or not brings in another challenge where we have to write more python code to be invoked from fpm, for logic that already exists and works in pip itself. I discovered however, that there is a pip install flag called --ignore-installed, which ignores already installed packages. The flag has a warning that it will overwrite installed packages which can break the python system, however this shouldn't be a problem for us since we are installing only into a temporary staging location. When I add this flag to our call to pip, all the tests pass in the CI without a virtual environment! See commits 70377dd and 3fd28b5.

I would say that the fact that we need a newer version of wheel is not a bug in fpm

because the new feature is for pyproject.toml Python projects, and this is a new feature in Python.

I'd like to understand more about this. It was not intended that older versions do not work.

I will address the wheel version incompatibility issue. To build a .whl file directly from a pyproject.toml project directory, you need wheel version 0.38.0 or greater (released 2022-10-21). This version replaced its internal use of the now deprecated distutils with the better setuptools package. Only the latter, setuptools, supports pyproject.toml. Ubuntu 22.04 comes with wheel version 0.37.1 which does not support pyproject.toml. Here is a reproduction of the problem in a Ubuntu 22.04 docker container, notice that it incorrectly names the package UNKNOWN-0.0.0 when using wheel version 0.37.1, but when using version 0.38.0 it correctly creates the package example-1.2.3:

### Ubuntu 22.04

$ python3 -c "from importlib.metadata import version; print(version('wheel'))"
0.37.1
$ pip install spec/fixtures/python-pyproject.toml/
Processing ./spec/fixtures/python-pyproject.toml
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: UNKNOWN
  Building wheel for UNKNOWN (pyproject.toml) ... done
  Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=961 sha256=0613a2297cc4ffd559af9d70c449dc76f4cb5ae4d2e3b25590ebefc3bb87fb98
  Stored in directory: /root/.cache/pip/wheels/d4/04/83/4a89942ae0272814aaae22b7fe389f2f426cb3f3ef96b67490
Successfully built UNKNOWN
Installing collected packages: UNKNOWN
  Attempting uninstall: UNKNOWN
    Found existing installation: UNKNOWN 0.0.0
    Uninstalling UNKNOWN-0.0.0:
      Successfully uninstalled UNKNOWN-0.0.0
Successfully installed UNKNOWN-0.0.0

$ pip install --upgrade wheel==0.38.0
$ python3 -c "from importlib.metadata import version; print(version('wheel'))"
0.38.0
$ pip install spec/fixtures/python-pyproject.toml
Processing ./spec/fixtures/python-pyproject.toml
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: example
  Building wheel for example (pyproject.toml) ... done
  Created wheel for example: filename=example-1.2.3-py3-none-any.whl size=967 sha256=4b58b2ac4f009ddca20bb14ef435903968adaddfca054583b6f59cbff0183a7a
  Stored in directory: /root/.cache/pip/wheels/d4/04/83/4a89942ae0272814aaae22b7fe389f2f426cb3f3ef96b67490
Successfully built example
Installing collected packages: example
Successfully installed example-1.2.3

This is why these tests fail in the CI for Expected "python-example", but got "python-unknown":

1) FPM::Package::Python when given a project containing a pyproject.toml should package it correctly
     Failure/Error: insist { subject.name } == "#{prefix}-example"

     Insist::Failure:
       Expected "python-example", but got "python-unknown"
     # ./vendor/bundle/ruby/3.1.0/gems/insist-1.0.0/lib/insist/assert.rb:8:in `assert'
     # ./vendor/bundle/ruby/3.1.0/gems/insist-1.0.0/lib/insist/comparators.rb:13:in `=='
     # ./spec/fpm/package/python_spec.rb:234:in `block (3 levels) in <top (required)>'

  2) FPM::Package::Python when given a project containing a pyproject.toml should package it correctly even if the path given is directly to the pyproject.toml
     Failure/Error: insist { subject.name } == "#{prefix}-example"

     Insist::Failure:
       Expected "python-example", but got "python-unknown"
     # ./vendor/bundle/ruby/3.1.0/gems/insist-1.0.0/lib/insist/assert.rb:8:in `assert'
     # ./vendor/bundle/ruby/3.1.0/gems/insist-1.0.0/lib/insist/comparators.rb:13:in `=='
     # ./spec/fpm/package/python_spec.rb:243:in `block (3 levels) in <top (required)>'

If I try myself on Ubuntu 22.04 in docker, python3 and wheel 0.37.1 (via dpkg/apt), packaging django fails:

This error you got (and that I was able to reproduce) is actually a new problem that I hadn't seen before. As you mentioned, the error comes from the lib/fpm/package/pyfpm/parse_requires.py script. I was able to fix the issue on Ubuntu 22.04 by passing environment={'extra':''} to the call to req.marker.evaluate(). However, in reading the documentation of packaging.markers, I learned that there is a function packaging.markers.default_environment() that returns a dict of the default environment of the current python process, and if I instead pass this dict as the value to environment= for the call to req.marker.evaluate() then everything works. See commit d1754cb for this change.

[wbraswell]

@jordansissel
As usual I have carefully reviewed everything with @NicholasBHubbard before posting here, and it all looks good to me.

[wbraswell]

Howdy @jordansissel , currently this is our most important PR because it will fix the broken CI and allow us to move forward with lots of other fixes accordingly. If you are short on time nowadays then you can simply accept and merge this PR as-is safely, or if you have a few minutes then you can also review the changes we made to the FPM code related to packaging Python projects. Either way, it should be safe for you to merge this PR now so that we can get un-stuck. Thanks in advance!

[jordansissel]

Noted! I’ll work on this as soon as I can. I appreciate the effort to stabilize github actions :)

…

On Thu, Dec 4, 2025, at 10:43 AM, William N. Braswell, Jr. wrote: *wbraswell* left a comment (jordansissel/fpm#2122) <#2122 (comment)> Howdy @jordansissel <https://github.com/jordansissel> , currently this is our most important PR because it will fix the broken CI and allow us to move forward with lots of other fixes accordingly. If you are short on time nowadays then you can simply accept and merge this PR as-is safely, or if you have a few minutes then you can also review the changes we made to the FPM code related to packaging Python projects. Either way, it should be safe for you to merge this PR now so that we can get un-stuck. Thanks in advance! — Reply to this email directly, view it on GitHub <#2122 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABAF2QQ4XMYFEJOUPY4LLD4AB6D5AVCNFSM6AAAAACLJC25N6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMMJTHAZDGNZZGU>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[wbraswell]

@jordansissel
Okay great thanks, please let us know if you find any problems otherwise it should be good to merge as-is. :-)