Need to rebuild wheels for every pypy minor version

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Need to rebuild wheels for every pypy minor version

Daniele Rolando
Hi guys.

Right now pypy wheels names include both the major and minor pypy version in them: e.g. uWSGI-2.0.14.0.pp257-pypy_41-linux_x86_64.whl
This means that if we want to upgrade pypy from 5.7.1 to 5.8 we'd need to rebuild all our wheels and this is not scalable since there are new pypy releases every 3/4 months.

Wouldn't it be enough to only include the major version in the wheel name? Are minor pypy versions really incompatible between them?

Thanks,
Daniele

_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: Need to rebuild wheels for every pypy minor version

Antonio Cuni
Hi,

note that this is not because of PyPy: it's the wheel package which chooses what to include in the wheel filename:

PyPy reports only the ABI version, which is pypy_41. This is probably wrong for the opposite reasons, i.e. it claims it's backward compatible even when it's not: https://bitbucket.org/pypy/pypy/issues/2613/fix-the-abi-tag

ciao,
Antonio

On Tue, Jul 25, 2017 at 8:47 PM, Daniele Rolando <[hidden email]> wrote:
Hi guys.

Right now pypy wheels names include both the major and minor pypy version in them: e.g. uWSGI-2.0.14.0.pp257-pypy_41-linux_x86_64.whl
This means that if we want to upgrade pypy from 5.7.1 to 5.8 we'd need to rebuild all our wheels and this is not scalable since there are new pypy releases every 3/4 months.

Wouldn't it be enough to only include the major version in the wheel name? Are minor pypy versions really incompatible between them?

Thanks,
Daniele

_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev



_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: Need to rebuild wheels for every pypy minor version

Nathaniel Smith
On Jul 25, 2017 12:16 PM, "Antonio Cuni" <[hidden email]> wrote:
Hi,

note that this is not because of PyPy: it's the wheel package which chooses what to include in the wheel filename:

Well, probably it is most correct to say that this happens because PyPy has not come up with a coherent strategy for ABI compatibility and communicated that to wheel/pip :-).

In the mean time, if you want to live dangerously, you can always rename your wheel something like 'foo-py2-pypy_41-linux_x86_64.whl', and then pip will happily install it on newer pypys. Whether it will work though is hard to say...

-n

_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: Need to rebuild wheels for every pypy minor version

Daniele Rolando
If I understand this correctly, the issue is that pip/wheel cannot rely on the ABI version from pypy so they have no way if a wheel would still be compatible or not. So they just generate a different wheel for every version to avoid having to deal with that problem at all.

Is that correct?

On Tue, Jul 25, 2017 at 1:24 PM Nathaniel Smith <[hidden email]> wrote:
On Jul 25, 2017 12:16 PM, "Antonio Cuni" <[hidden email]> wrote:
Hi,

note that this is not because of PyPy: it's the wheel package which chooses what to include in the wheel filename:

Well, probably it is most correct to say that this happens because PyPy has not come up with a coherent strategy for ABI compatibility and communicated that to wheel/pip :-).

In the mean time, if you want to live dangerously, you can always rename your wheel something like 'foo-py2-pypy_41-linux_x86_64.whl', and then pip will happily install it on newer pypys. Whether it will work though is hard to say...

-n

_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: Need to rebuild wheels for every pypy minor version

matti picus
In reply to this post by Antonio Cuni
Both the ABI version identifier pypy_41 and the python identifier pp257
need overhauling. I have submitted issues to pip and wheel for the
Python Tag pp257 (i.e. pypy python 2 version 5.7) which IMO should be
pp27 (i.e. pypy implementing python 2.7).
https://github.com/pypa/pip/issues/4631
https://bitbucket.org/pypa/wheel/issues/187

Matti

On 25/07/17 22:15, Antonio Cuni wrote:

> Hi,
>
> note that this is not because of PyPy: it's the wheel package which
> chooses what to include in the wheel filename:
> https://bitbucket.org/pypa/wheel/src/5d49f8cf18679d1bc6f3e1e414a5df3a1e492644/wheel/pep425tags.py?at=default&fileviewer=file-view-default#pep425tags.py-39
>
> PyPy reports only the ABI version, which is pypy_41. This is probably
> wrong for the opposite reasons, i.e. it claims it's backward
> compatible even when it's not:
> https://bitbucket.org/pypy/pypy/issues/2613/fix-the-abi-tag
>
> ciao,
> Antonio
>
> On Tue, Jul 25, 2017 at 8:47 PM, Daniele Rolando
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Hi guys.
>
>     Right now pypy wheels names include both the major and minor pypy
>     version in them: e.g. uWSGI-2.0.14.0.*pp257*-pypy_41-linux_x86_64.whl
>     This means that if we want to upgrade pypy from 5.7.1 to 5.8 we'd
>     need to rebuild all our wheels and this is not scalable since
>     there are new pypy releases every 3/4 months.
>
>     Wouldn't it be enough to only include the major version in the
>     wheel name? Are minor pypy versions really incompatible between them?
>
>     Thanks,
>     Daniele
>
>     _______________________________________________
>     pypy-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/pypy-dev
>     <https://mail.python.org/mailman/listinfo/pypy-dev>
>
>
>
>
> _______________________________________________
> pypy-dev mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/pypy-dev

_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: Need to rebuild wheels for every pypy minor version

Nathaniel Smith
I think that the intended idea is:

The first field says what variant of the Python language the wheel
uses. So it can be py2, py33, etc. for generic "python 2", "python 3.3
or better". Or it can be cp33 to mean "we specifically need cpython
3.3, other language variants won't do".

And then the second field says what variant of the C ABI the wheel
uses. So it can be cp35dm to mean "CPython, 3.5, debug, pymalloc".

In practice this distinction doesn't make a *ton* of sense for CPython
wheels, because it's redundant -- pretty much the only reason why you
might need the cpython implementation specifically is if you're using
the C ABI, and if you need the cp35dm ABI then obviously only CPython
can give you that.

bdist_wheel by default uses both, because, uh... belt *and*
suspenders, I guess? But in principle they could vary independently.
For example, if some package is python 2, and depends on some API in
__pypy__, but doesn't use cpyext, then it might make sense to tag that
as 'pp2_57-none' or something to mean "I need the pypy 2
v5.7-or-better variant of the language, but I don't care about the
ABI".

In general pypy can come up with any system that makes sense here. A
given release can tell pip that it's compatible with whatever mix of
python tags and ABI tags that make sense, and bdist_wheel can have
whatever defaults make sense as well (and in general users can
override if they have a weird case). You could even rethink it a bit
and make extension modules generally use tags like 'py27-pp2_58' ("the
python part is vanilla python 2.7, but it also needs the pypy2 v5.8
ABI"), and you could have a single release of PyPy declare that it
supports multiple ABIs (e.g. you could have a pp2_cffi${X} ABI that
you keep relatively stable and increment occasionally, as well as a
pp2_cpyext${X} ABI that gets revved more often.)

I guess one thing is that the python tags and ABI tags should probably
be distinct for the pypy2 and pypy3 branches. And a "pp27" tag doesn't
make much sense to me, because the difference between pp27 and py27
would be that the former declares that it specifically needs some kind
of pypy extension... but what version of pypy added that extension?
There's no way to say.

Anyway, I guess the main thing is to think ahead about what kind of
ABI/stability/evolution strategy you actually want to use :-).

-n

On Tue, Jul 25, 2017 at 2:30 PM, Matti Picus <[hidden email]> wrote:

> Both the ABI version identifier pypy_41 and the python identifier pp257 need
> overhauling. I have submitted issues to pip and wheel for the Python Tag
> pp257 (i.e. pypy python 2 version 5.7) which IMO should be pp27 (i.e. pypy
> implementing python 2.7).
> https://github.com/pypa/pip/issues/4631
> https://bitbucket.org/pypa/wheel/issues/187
>
> Matti
>
> On 25/07/17 22:15, Antonio Cuni wrote:
>>
>> Hi,
>>
>> note that this is not because of PyPy: it's the wheel package which
>> chooses what to include in the wheel filename:
>>
>> https://bitbucket.org/pypa/wheel/src/5d49f8cf18679d1bc6f3e1e414a5df3a1e492644/wheel/pep425tags.py?at=default&fileviewer=file-view-default#pep425tags.py-39
>>
>> PyPy reports only the ABI version, which is pypy_41. This is probably
>> wrong for the opposite reasons, i.e. it claims it's backward compatible even
>> when it's not: https://bitbucket.org/pypy/pypy/issues/2613/fix-the-abi-tag
>>
>> ciao,
>> Antonio
>>
>> On Tue, Jul 25, 2017 at 8:47 PM, Daniele Rolando
>> <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>     Hi guys.
>>
>>     Right now pypy wheels names include both the major and minor pypy
>>     version in them: e.g. uWSGI-2.0.14.0.*pp257*-pypy_41-linux_x86_64.whl
>>     This means that if we want to upgrade pypy from 5.7.1 to 5.8 we'd
>>     need to rebuild all our wheels and this is not scalable since
>>     there are new pypy releases every 3/4 months.
>>
>>     Wouldn't it be enough to only include the major version in the
>>     wheel name? Are minor pypy versions really incompatible between them?
>>
>>     Thanks,
>>     Daniele
>>
>>     _______________________________________________
>>     pypy-dev mailing list
>>     [hidden email] <mailto:[hidden email]>
>>     https://mail.python.org/mailman/listinfo/pypy-dev
>>     <https://mail.python.org/mailman/listinfo/pypy-dev>
>>
>>
>>
>>
>> _______________________________________________
>> pypy-dev mailing list
>> [hidden email]
>> https://mail.python.org/mailman/listinfo/pypy-dev
>
>
> _______________________________________________
> pypy-dev mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/pypy-dev



--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev