| Age | Commit message (Collapse) | Author |
|---|
|
This commit, together with https://github.com/NixOS/nixpkgs/pull/442540,
changes the way python environments are built:
* When generating wrappers for python executables, we inherit argv[0]
from the wrapper. This causes python to initialize its configuration
in the environment with all the correct paths.
* We also resolve argv[0] to absolute path when invoking python from
PATH. This helps set python's prefix correctly on Darwin.
The end result is that python environments no longer appear to be venvs,
and behave more like a vanilla python installation. In addition it's
possible to create a venv using an environment and use packages from
both the environment and the venv.
|
|
|
|
|
|
Format all Nix files using the officially approved formatter,
making the CI check introduced in the previous commit succeed:
nix-build ci -A fmt.check
This is the next step of the of the [implementation](https://github.com/NixOS/nixfmt/issues/153)
of the accepted [RFC 166](https://github.com/NixOS/rfcs/pull/166).
This commit will lead to merge conflicts for a number of PRs,
up to an estimated ~1100 (~33%) among the PRs with activity in the past 2
months, but that should be lower than what it would be without the previous
[partial treewide format](https://github.com/NixOS/nixpkgs/pull/322537).
Merge conflicts caused by this commit can now automatically be resolved while rebasing using the
[auto-rebase script](https://github.com/NixOS/nixpkgs/tree/8616af08d915377bd930395f3b700a0e93d08728/maintainers/scripts/auto-rebase).
If you run into any problems regarding any of this, please reach out to the
[formatting team](https://nixos.org/community/teams/formatting/) by
pinging @NixOS/nix-formatting.
|
|
This reverts commit 96118850f323f41c163f1f5a8231128879fb2f2e.
|
|
When building a python environment's bin directory, we now detect
wrapped python scripts from installed packages, and generate unwrapped
copies with the environment's python executable as the interpreter.
|
|
|
|
A "python" made with the wrapper is likely to be used as a shebang. On macOS,
this requires a binary rather than another shebang'd script.
|
|
Now that `buildEnv` is ready, always put `makeWrapper` in
`nativeBuildInputs`, rather than `buildInputs` or (worse) mucking around
with setup hooks by hand.
(C.f. #112276, which didn't catch these because the manual setup hook
sourcing is such a hack to being with!)
Fixes #114687
|
|
|
|
This is needed in case of `python.buildEnv` to make sure site.PREFIXES
does not only point to the unwrapped executable prefix.
--------------------------------------------------------------------------------
This PR is a story where your valiant hero sets out on a very simple adventure but ends up having to slay dragons, starts questioning his own sanity and finally manages to gain enough knowledge to slay the evil dragon and finally win the proverbial price.
It all started out on sunny spring day with trying to tackle the Nixops plugin infrastructure and make that nice enough to work with.
Our story begins in the shanty town of [NixOps-AWS](https://github.com/nixos/nixops-aws) where [mypy](http://mypy-lang.org/) type checking has not yet been seen.
As our deuteragonist (@grahamc) has made great strides in the capital city of [NixOps](https://github.com/nixos/nixops) our hero wanted to bring this out into the land and let the people rejoice in reliability and a wonderful development experience.
The plugin work itself was straight forward and our hero quickly slayed the first small dragon, at this point things felt good and our hero thought he was going to reach the town of NixOps-AWS very quickly.
But alas! Mypy did not want to go, it said:
`Cannot find implementation or library stub for module named 'nixops'`
Our hero felt a small sliver of life escape from his body. Things were not going to be so easy.
After some frustration our hero discovered there was a [rule of the land of Python](https://www.python.org/dev/peps/pep-0561/) that governed the import of types into the kingdom, more specificaly a very special document (file) called `py.typed`.
Things were looking good.
But no, what the law said did not seem to match reality. How could things be so?
After some frustrating debugging our valiant hero thought to himself "Hmm, I wonder if this is simply a Nix idiosyncrasy", and it turns out indeed it was.
Things that were working in the blessed way of the land of Python (inside a `virtualenv`) were not working the way they were from his home town of Nix (`nix-shell` + `python.withPackages`).
After even more frustrating attempts at reading the mypy documentation and trying to understand how things were supposed to work our hero started questioning his sanity.
This is where things started to get truly interesting.
Our hero started to use a number of powerful weapons, both forged in the land of Python (pdb) & by the mages of UNIX (printf-style-debugging & strace).
After first trying to slay the dragon simply by `strace` and a keen eye our hero did not spot any weak points.
Time to break out a more powerful sword (`pdb`) which also did not divulge any secrets about what was wrong.
Our hero went back to the `strace` output and after a fair bit of thought and analysis a pattern started to emerge. Mypy was looking in the wrong place (i.e. not in in the environment created by `python.withPackages` but in the interpreter store path) and our princess was in another castle!
Our hero went to the pub full of old grumpy men giving out the inner workings of the open source universe (Github) and acquired a copy of Mypy.
He littered the code with print statements & break points.
After a fierce battle full of blood, sweat & tears he ended up in https://github.com/python/mypy/blob/20f7f2dd71c21bde4d3d99f9ab69bf6670c7fa03/mypy/sitepkgs.py and realised that everything came down to the Python `site` module and more specifically https://docs.python.org/3.7/library/site.html#site.getsitepackages which in turn relies on https://docs.python.org/3.7/library/site.html#site.PREFIXES .
Our hero created a copy of the environment created by `python.withPackages` and manually modified it to confirm his findings, and it turned out it was indeed the case.
Our hero had damaged the dragon and it was time for a celebration.
He went out and acquired some mead which he ingested while he typed up his story and waited for the dragon to finally die (the commit caused a mass-rebuild, I had to wait for my repro).
In the end all was good in [NixOps-AWS](https://github.com/nixos/nixops-aws)-town and type checks could run. (PR for that incoming tomorrow).
|
|
This is needed in case of `python.buildEnv` to make sure sys.executable
does not point to the unwrapped executable.
|
|
|
|
|
|
`python.buildEnv` would already wrap executables exporting `PYTHONHOME`.
With this change, it is possible to pass in additional arguments to the
underlying `makeWrapper`.
|
|
28299f669adc41e5278372cad952fb1e1165b44b introduced the first Python
packages having multiple outputs. The required outputs were not picked
up by `python.buildEnv` (#31857).
This commit modifies `python.buildEnv` so that it always includes the
$out output and thus fixes #31857.
|
|
Python libraries or modules now have an attribute `pythonModule = interpreter;` to indicate
they provide Python modules for the specified `interpreter`.
The package set provides the following helper functions:
- hasPythonModule: Check whether a derivation provides a Python module.
- requiredPythonModules: Recurse into a list of Python modules, returning all Python modules that are required.
- makePythonPath: Create a PYTHONPATH from a list of Python modules.
Also included in this commit is:
- disabledIf: Helper function for disabling non-buildPythonPackage functions.
|
|
|
|
|
|
|
|
Python envs did not pass through any of the properties the Python
interpreter has. That could be annoying, especially not having
`python.interpreter` which is the path to the interpreter. This commit
fixes the situation and inherit python.passthru.
|
|
Python by default checks a `site-packages` folder in the user's home
folder. We do not want such an impurity and therefore disable it.
Fixes #26846.
|
|
python.buildenv is used to build an env that provides binaries that can
import all modules that were passed in to the env.
Before this change it filtered the propagatedBuildInputs to remove all
non-Python packages, thereby possibly reducing the amount of packages
that were referenced. However, Python packages often don't have non-
Python packages as propagatedBuildInputs. And occasionally, we do want
to be able to add other packages to the env.
|
|
Currently, when constructing a buildEnv and adding packages via
extraLibs, then binaries in extraLibs cannot access the other Python
modules. An example is having ipython/jupyter in extraLibs; in that case
ipython cannot import any other modules.
|
|
|
|
|
|
|
|
|
|
Didn't mean to commit this change
Signed-off-by: Shea Levy <shea@shealevy.com>
|
|
Signed-off-by: Shea Levy <shea@shealevy.com>
|
|
'postBuild' step
The default setting for extraLibs used to be the set of modules that come with
python by default but aren't usually enabled in our standard python derivation
because they require additional libraries. This meant that users who want to
*add* libraries to that set had to use a fairly complicated override, to add
more entries without loosing the ones set by default.
After this patch, the "standard libraries" such as "curses' are listed in
stdLibs while the extraLibs argument remains empty by default. This allows
users to override extraLibs without overriding the standard libraries.
Furthermore, the wrapper environment can be messed around with in an
additional 'postBuild' step. One nice application of this build step is
to patch scripts and binaries to use the wrapped python interpreter
instead of the pristine one, thereby enabling them to pick up all
modules that have been configured. The following example shows how this
is done for the 'pylint' utility:
pkgs.python27Full.override {
extraLibs = [pkgs.pylint];
postBuild = ''
cd ${pkgs.pylint}/bin
for i in *; do
rm $out/bin/$i
sed -r -e "s|^exec |exec $out/bin/python -- |" <$i >$out/bin/$i
chmod +x $out/bin/$i
done;
'';
};
|
|
'extraLibs' in the generated environment
This patch means that adding 'matplotlib' to extraLibs will automatically
include 'numpy', too, because matplotlib depends on it.
|
|
See discussion in #834
Signed-off-by: Shea Levy <shea@shealevy.com>
|
|
The new wrapper creates an environment that contains all files from
Python and the extra libraries that have been specified. All files are
found at run-time by means of the $PYTHONHOME variable; the wrapper no
longer uses $PYTHONPATH.
|
|
|
|
|
|
|
|
This reverts commit 3ee2667e4c60c2ed850da8538cf135fe7f716f30.
svn path=/nixpkgs/branches/stdenv-updates/; revision=32656
|
|
This reverts commit f77b9a16a9ef52951a601997593dc557a42660b9.
svn path=/nixpkgs/branches/stdenv-updates/; revision=32653
|
|
svn path=/nixpkgs/branches/stdenv-updates/; revision=32593
|
|
svn path=/nixpkgs/branches/stdenv-updates/; revision=32589
|
|
svn path=/nixpkgs/branches/stdenv-updates/; revision=32588
|
|
svn path=/nixpkgs/branches/stdenv-updates/; revision=32587
|
|
function, so obsolete it.
svn path=/nixpkgs/branches/stdenv-updates/; revision=31644
|
|
svn path=/nixpkgs/trunk/; revision=23200
|
|
The python wrapper expression expects a list of Python modules, $extraLibs,
which are added to $PYTHONPATH before executing the actual Python interpreter.
svn path=/nixpkgs/trunk/; revision=23194
|
