okay to rename cppyy -> _cppyy

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

okay to rename cppyy -> _cppyy

wlavrijsen@lbl.gov
Hi,

any objections to renaming cppyy into _cppyy?

I want to be able to do a straight 'pip install cppyy' and then use it
w/o further gymnastics (this works today for CPython), but then I can't
have 'cppyy' be a built-in module.

(You can pip install PyPy-cppyy-backend, but then you'd still have to deal
with LD_LIBRARY_PATH and certain cppyy features are PyPy version dependent
even as they need not be as they are pure Python.)

The pip-installed cppyy will still use the built-in _cppyy for the PyPy
specific parts (low-level manipulations etc.).

I'm also moving the cppyy documentation out of the pypy documentation and
placing it on its own (http://cppyy.readthedocs.io/), given that the
CPython side of things now works, too.

Yes, no, conditional?

Thanks,
      Wim
--
[hidden email]    --    +1 (510) 486 6411    --    www.lavrijsen.net
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: okay to rename cppyy -> _cppyy

Maciej Fijalkowski
that's how cffi works FYI. +1 from me

On Wed, Jul 19, 2017 at 12:33 AM,  <[hidden email]> wrote:

> Hi,
>
> any objections to renaming cppyy into _cppyy?
>
> I want to be able to do a straight 'pip install cppyy' and then use it
> w/o further gymnastics (this works today for CPython), but then I can't
> have 'cppyy' be a built-in module.
>
> (You can pip install PyPy-cppyy-backend, but then you'd still have to deal
> with LD_LIBRARY_PATH and certain cppyy features are PyPy version dependent
> even as they need not be as they are pure Python.)
>
> The pip-installed cppyy will still use the built-in _cppyy for the PyPy
> specific parts (low-level manipulations etc.).
>
> I'm also moving the cppyy documentation out of the pypy documentation and
> placing it on its own (http://cppyy.readthedocs.io/), given that the
> CPython side of things now works, too.
>
> Yes, no, conditional?
>
> Thanks,
>      Wim
> --
> [hidden email]    --    +1 (510) 486 6411    --    www.lavrijsen.net
> _______________________________________________
> pypy-dev mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/pypy-dev
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: okay to rename cppyy -> _cppyy

Omer Katz
+1. Great work!

‫בתאריך יום ד׳, 19 ביולי 2017 ב-0:58 מאת ‪Maciej Fijalkowski‬‏ <‪[hidden email]‬‏>:‬
that's how cffi works FYI. +1 from me

On Wed, Jul 19, 2017 at 12:33 AM,  <[hidden email]> wrote:
> Hi,
>
> any objections to renaming cppyy into _cppyy?
>
> I want to be able to do a straight 'pip install cppyy' and then use it
> w/o further gymnastics (this works today for CPython), but then I can't
> have 'cppyy' be a built-in module.
>
> (You can pip install PyPy-cppyy-backend, but then you'd still have to deal
> with LD_LIBRARY_PATH and certain cppyy features are PyPy version dependent
> even as they need not be as they are pure Python.)
>
> The pip-installed cppyy will still use the built-in _cppyy for the PyPy
> specific parts (low-level manipulations etc.).
>
> I'm also moving the cppyy documentation out of the pypy documentation and
> placing it on its own (http://cppyy.readthedocs.io/), given that the
> CPython side of things now works, too.
>
> Yes, no, conditional?
>
> Thanks,
>      Wim
> --
> [hidden email]    --    <a href="tel:+1%20510-486-6411" value="+15104866411" target="_blank">+1 (510) 486 6411    --    www.lavrijsen.net
> _______________________________________________
> pypy-dev mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/pypy-dev
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev

_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: okay to rename cppyy -> _cppyy

Antonio Cuni
In reply to this post by wlavrijsen@lbl.gov
+1

On Tue, Jul 18, 2017 at 11:33 PM, <[hidden email]> wrote:
Hi,

any objections to renaming cppyy into _cppyy?

I want to be able to do a straight 'pip install cppyy' and then use it
w/o further gymnastics (this works today for CPython), but then I can't
have 'cppyy' be a built-in module.

(You can pip install PyPy-cppyy-backend, but then you'd still have to deal
with LD_LIBRARY_PATH and certain cppyy features are PyPy version dependent
even as they need not be as they are pure Python.)

The pip-installed cppyy will still use the built-in _cppyy for the PyPy
specific parts (low-level manipulations etc.).

I'm also moving the cppyy documentation out of the pypy documentation and
placing it on its own (http://cppyy.readthedocs.io/), given that the
CPython side of things now works, too.

Yes, no, conditional?

Thanks,
     Wim
--
[hidden email]    --    <a href="tel:%2B1%20%28510%29%20486%206411" value="+15104866411" target="_blank">+1 (510) 486 6411    --    www.lavrijsen.net
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev


_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: okay to rename cppyy -> _cppyy

wlavrijsen@lbl.gov
In reply to this post by Maciej Fijalkowski
Maciej,

yes, I know. :) I used and use CFFI for inspiration.

Now done. The package structure has become:

                       _cppyy for PyPy
                   /                      \
    cppyy (shared)                          cppyy-backend (shared)
                   \                      /
                     CPyCppyy for CPython

For pip-purposes, the cppyy wheel is not shared (but it's cheap to install).
The backend bdist_wheel can be re-used for PyPy, CPython2, and CPython3.

  $ virtualenv -p .../pypy/pypy/goal/pypy-c pypy-dev
  $ MAKE_NPROCS=32 pip install cppyy
  ...
  $ python
  Python 2.7.13 (0b40d2587588, Jul 19 2017, 00:43:13)
  [PyPy 5.9.0-alpha0 with GCC 5.4.0] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  using my private settings ...
  And now for something completely different: ``Therefore, specific information,
  I was in an ideal context, I had to realize the faith''
  >>>> import cppyy
  >>>> cppyy.cppdef('void say_hello() { cout << "Hello, World!" << endl; }')
  >>>> cppyy.gbl.say_hello()
  Hello, World!
  >>>>

The documentation and (shared) app-level tests live in cppyy. I have some
more work to do there.

For PyPy-5.7 and -5.8, this can be done:
  >>>> from cppyy_backend import loader
  >>>> c = loader.load_cpp_backend()
  >>>> import cppyy

and things work from there (albeit not niceties such as cppdef() above, but
all the genreflex etc. stuff works).

Omer, Anto, thanks for pushing. :) I'm actually pretty pleased with this
setup: it all works nicer than I expected. Life being what it is, the last
time I looked into distutils was apparently Oct 9, 2007 (I just checked my
archives). Things have improved quite a bit since then ...

Best regards,
            Wim
--
[hidden email]    --    +1 (510) 486 6411    --    www.lavrijsen.net
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: okay to rename cppyy -> _cppyy

Manuel Jacob
In reply to this post by wlavrijsen@lbl.gov
Sounds like a good idea.  Since this is similar to what is done with
cffi, I have a few questions:

How do you make sure that the pure Python part remains compatible with
the backend?

Can we include the pure Python part in PyPy, in the way as it's done for
cffi?

On 2017-07-18 23:33, [hidden email] wrote:

> Hi,
>
> any objections to renaming cppyy into _cppyy?
>
> I want to be able to do a straight 'pip install cppyy' and then use it
> w/o further gymnastics (this works today for CPython), but then I can't
> have 'cppyy' be a built-in module.
>
> (You can pip install PyPy-cppyy-backend, but then you'd still have to
> deal
> with LD_LIBRARY_PATH and certain cppyy features are PyPy version
> dependent
> even as they need not be as they are pure Python.)
>
> The pip-installed cppyy will still use the built-in _cppyy for the PyPy
> specific parts (low-level manipulations etc.).
>
> I'm also moving the cppyy documentation out of the pypy documentation
> and
> placing it on its own (http://cppyy.readthedocs.io/), given that the
> CPython side of things now works, too.
>
> Yes, no, conditional?
>
> Thanks,
>      Wim
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: okay to rename cppyy -> _cppyy

wlavrijsen@lbl.gov
Manuel,

> How do you make sure that the pure Python part remains compatible with the
> backend?

I'm thinking of selecting the versions explicitly in the front cppyy package:
it already has different dependencies selected based on interpreter: PyPy or
CPython. Also, the backend "API" is small and stable, so I expect a large
number of versions to work well, including older ones, moving forward.

If you have a better idea, then I'm all ears! I'm new to this packaging ...

> Can we include the pure Python part in PyPy, in the way as it's done for
> cffi?

Yes, albeit that it is still necessary to install cppyy_backend (contains a
patched version of Clang/LLVM), so I don't see much net gain.

And yes, it is technically possible to write a bindings generator that only
depends on LLVM during offline bindings generation, not at run-time. But
then you'd just have SWIG (albeit with a standards-compliant parser).

Best regards,
            Wim
--
[hidden email]    --    +1 (510) 486 6411    --    www.lavrijsen.net
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: okay to rename cppyy -> _cppyy

Armin Rigo-2
In reply to this post by wlavrijsen@lbl.gov
Hi Wim,

On 18 July 2017 at 23:33,  <[hidden email]> wrote:
> I want to be able to do a straight 'pip install cppyy' and then use it
> w/o further gymnastics (this works today for CPython)

Great, I'm +1 on that as well.

If the newly-named '_cppyy' module is more minimal than
'_cffi_backend' and less likely to change, then yes, I think it's a
good plan to include only '_cppyy' with PyPy and distribute the rest
with 'pip install'.


A bientôt,

Armin
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: okay to rename cppyy -> _cppyy

wlavrijsen@lbl.gov
Armin,

> If the newly-named '_cppyy' module is more minimal than '_cffi_backend'

'more minimal' is a hard to define term, but a 'wc -l' shows that
_cffi_backend has 8498 lines of python and _cppyy has 4265. Of course, the
latter uses the former, saving lots of code for it. :)

Further, _cppyy includes 470 lines in pythonify.py which could move out, as
it is not RPython, but that would make it harder to write tests, which would
have to be low-level (as in test_cppyy.py) instead of high level (the rest).

I wrote some small benchmarks for the PyHPC '16 paper. If I find the time
to flesh that out further, I can better judge what can move out of RPython
(_cppyy) and into Python (cppyy), without loss of performance.

Best regards,
            Wim
--
[hidden email]    --    +1 (510) 486 6411    --    www.lavrijsen.net
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: okay to rename cppyy -> _cppyy

wlavrijsen@lbl.gov
In reply to this post by wlavrijsen@lbl.gov
Hi,

[replying to self]

> And yes, it is technically possible to write a bindings generator that only
> depends on LLVM during offline bindings generation, not at run-time. But
> then you'd just have SWIG (albeit with a standards-compliant parser).

it hit me yesterday that instead of generating something offline for cppyy
to read, you could also use cppyy_backend directly to generate a pure python
module for use with cffi as it knows about sizes, offsets, name mangling,
class relations etc., etc., etc.

(The point is that cppyy_backend exposes a minimalistic C-API, which itself
can be bound with cffi, as _cppyy in fact does.)

(There are still wrappers needed for inlined functions, overloaded
operator new, virtual inheritance offsets, etc., but those could be stuffed
in a .cxx on the side of the cffi-based Python module. Can't solve automatic
template instantiation this way, though: cppyy will always be superior.)

So, I wrote a quick proof of concept (https://bitbucket.org/wlav/quaff/src):

  $ cat Simple.h
  class Simple {
  public:
      Simple();
      Simple(int);
      virtual ~Simple();

  public:
      int get_data();
      void set_data(int);

  private:
      int m_data;
  };

  $ cat Simple.cxx
  #include "Simple.h"

  Simple::Simple() : m_data(42) { }
  Simple::Simple(int i) : m_data(i) { }
  Simple::~Simple() { }

  int Simple::get_data() { return m_data; }
  void Simple::set_data(int i) { m_data = i; }

  $ g++ -shared -o libsimple.so -fPIC Simple.cxx

  $ python
  >>>> from quaff import generate
  >>>> generate.generate_class('simple', 'Simple', 'libsimple.so')
  >>>> import simple
  >>>> s1 = simple.Simple()
  >>>> s1.get_data()
  42
  >>>> s1.set_data(13)
  >>>> s1.get_data()
  13
  >>>> s2 = simple.Simple(99)
  >>>> s2.get_data()
  99
  >>>>

What quaff generated looks like this:

  $ cat simple.py
  from cffi import FFI

  _ffi = FFI()

  _ffi.cdef("""
  void _ZN6SimpleC1Ev(void*);
  void _ZN6SimpleC1Ei(void*, int);
  int _ZN6Simple8get_dataEv(void*);
  void _ZN6Simple8set_dataEi(void*, int);
  """)

  _dll = _ffi.dlopen('libsimple.so')

  class Simple(object):
    def __init__(self, *args):
      self._cpp_this = _ffi.new('char[]', 16)
      if len(args) == 0:
        _dll._ZN6SimpleC1Ev(self._cpp_this, *args)
      if len(args) == 1:
        _dll._ZN6SimpleC1Ei(self._cpp_this, *args)
    def get_data(self, *args):
      return _dll._ZN6Simple8get_dataEv(self._cpp_this, *args)
    def set_data(self, *args):
      return _dll._ZN6Simple8set_dataEi(self._cpp_this, *args)

which is nicely standalone for distribution (yes, the result is platform
specific, but no more so than a binary .so and way better than a Python
extension module).

Best regards,
            Wim
--
[hidden email]    --    +1 (510) 486 6411    --    www.lavrijsen.net
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: okay to rename cppyy -> _cppyy

Armin Rigo-2
Hi Wim,

On 21 July 2017 at 03:17,  <[hidden email]> wrote:
>  _ffi.cdef("""
>  void _ZN6SimpleC1Ev(void*);
>  void _ZN6SimpleC1Ei(void*, int);
>  int _ZN6Simple8get_dataEv(void*);
>  void _ZN6Simple8set_dataEi(void*, int);
>  """)

Ouch, I suppose :-)  Explicit C++ mangling.  The alternative would be
to use ffi.set_source() instead of ffi.dlopen(), and use a C++
compiler again to produce the cffi binding.  This reduces cppyy to a
generator emitting a cffi build script.  But as you mentioned there
are additional issues, e.g. with instantiation of templates.


A bientôt,

Armin.
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: okay to rename cppyy -> _cppyy

wlavrijsen@lbl.gov
Armin,

> Ouch, I suppose :-)  Explicit C++ mangling.  The alternative would be
> to use ffi.set_source() instead of ffi.dlopen(), and use a C++
> compiler again to produce the cffi binding.

sure, but the mangled name saves a redirection and has less code to generate,
was my point. Besides, who looks at generated code. :) Have you seen what
comes out of the PyPy translator ...

> But as you mentioned there are additional issues, e.g. with instantiation
> of templates.

And all other dynamic behaviors, incl. deriving Python classes from C++
ones for use in C++-land. (Is currently disabled, as I never implemented
that for the PyPy side, only CPython (it uses the Python C-API).) Or
general patch-ups (e.g. byte* <-> char*). Or auto-downcasting (favorite).
Or class unique-ness. Or adding user-defined classes interactively.

As said, in the end it'd be just SWIG but with a better parser (and no
Python versioning issues, or new language to learn).

I'd like to see the performance of a more complex case, though. E.g. when
doing class matches using Python reflection such as isinstance. I've always
had a hard time getting these things fast in RPython, and PyPy may have an
easier time with proper Python code (another reason to reject C++ wrappers
and live with mangled names).

Best regards,
            Wim
--
[hidden email]    --    +1 (510) 486 6411    --    www.lavrijsen.net
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: okay to rename cppyy -> _cppyy

Armin Rigo-2
Hi Wim,

On 21 July 2017 at 09:46,  <[hidden email]> wrote:
> had a hard time getting these things fast in RPython, and PyPy may have an
> easier time with proper Python code (another reason to reject C++ wrappers
> and live with mangled names).

Comparing ABI mode (ffi.dlopen) and API mode (ffi.set_source) for
performance, people often fail to realize that the API mode is
generally much faster because calls don't go through libffi.  That is
always the case on CPython.

On PyPy, the JIT can compile all calls in API mode, but only "simple
enough" calls in ABI mode, making performance more predictable.  For
the cases where the call is "simple enough", you might think that the
API mode adds the marginal cost of a wrapper, but that is mostly wrong
too: the JIT replaces a call to a JMP instruction with a call to the
jmp target.


A bientôt,

Armin.
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev
Reply | Threaded
Open this post in threaded view
|

Re: okay to rename cppyy -> _cppyy

wlavrijsen@lbl.gov
Armin,

> people often fail to realize

count me among those. :) Yes, that is surprising.

> For the cases where the call is "simple enough", you might think that the
> API mode adds the marginal cost of a wrapper

To be sure, if the code is really simple, it's typically inlined on the C++
end and then a wrapper is needed anyway (shouldn't, but linkers tends to be
too smart by half as we found out).

Alright, let me try some micro-benches. If it can outperform CPython/cppyy
then I'm interested in pursuing this track. (With cppyy, wrappers are already
generated anyway and compiled by the LLVM JIT, but those are fully generic
ones, leaving lots of (un)packing and some copying on each call.)

Best regards,
            Wim
--
[hidden email]    --    +1 (510) 486 6411    --    www.lavrijsen.net
_______________________________________________
pypy-dev mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/pypy-dev