Big changes planned for CEF Python

[cztomczak]

There will be lots of enhancements, new features, but unfortunately there will also be a huge refactoring of API and that will break existing applications. We will provide a Migration Guide though. A taste of what is to come is outlined here: https://github.com/cztomczak/pycef/blob/master/README.md#pycef

Goals for the near future

• automate.py - Automate building of CEF with CEF Python patches applied
• 51.0 beta release for Linux / Python 2.7
• Create examples/ root dir - simplify and unify examples to work on all platforms - Issue Create examples/ root dir - simplify and unify examples to work on all platforms #207
• Fix issues labeled with bug
• Use CMake on all platforms - Issue Create tools/build.py. Automate building C++ libs and subprocess and make it cross-platform. Windows platform already supported. #210
• New build tools in the tools/ directory that work cross-platform (build.py, make-setup.py)
• Update to the latest Cython 0.24 - Issue Update Cython to latest #110
• Python 3 support - Issue Support for Python 3.4 / 3.5 / 3.6 #121 and Issue Support for Python 3.5 #183
• translator.py - Expose as much of CEF API in an automated way as possible (Issue Create translator.py tool to provide full CEF API coverage in an automated way  #214)
  • start with easy stuff: enum types - expose constants and Enum objects and put them in the cefpython3.types submodule
  • client handlers and callbacks - generate .pyx and .h/c++ files

Plan for progressive enhancements

It is a huge task to make all these changes. It can take a long time and it can break lots of things, so a good plan is needed so that we don't end up with broken code and a demotivated mind. Divide work into small goals - small tasks - and check them when done, so you can see the progress. Make a list of all the stuff that needs to be done and the order in which it needs to be refactored - so that you can plan ahead to predict issues and have a good design that doesn't change on the fly while refactoring - that would be disastrous to end up with broken stuff and seeing no near future in having it fixed.

1. Create new branch "dev"?
2. Refactor build tools. Move to tools/. Examples will be refactored to allow running only as a package.
   • refactor .pyx files - use them as real modules? get rid of the include statements?
   • rename the package from cefpython3 to "cefpython"
   • build.py - build the package
   • install.py - install the package
   • run.py - build install the package automatically
   • use Cmake on Windows
3. Create unit tests.
   • start with tests for the core / helloworld example: init, run, stop, work, quit, Browser.
   • add unit test for each function/class that is being refactored
4. Create examples/: helloworld.py and others
5. Unicode strings in Py2 - this is a big BC break so PyPI package name should be changed eg. from cefpython3 to cefpython
6. API refactoring: pep8, submodules (app, browser, interfaces) - as above in regards to package name
7. Automate creation of bindings to CEF with the translator.py tool - Issue Create translator.py tool to provide full CEF API coverage in an automated way  #214. See also Issue ApplicationSettings and BrowserSettings as classes #205 (ApplicationSettings and BrowserSettings as classes) and Issue Expose handlers as classes/interfaces: ClientHandler, DisplayHandler, LifespanHandler, etc. #215 (Expose handlers as classes/interfaces: ClientHandler, DisplayHandler, LifespanHandler, etc).
8. Documentation:
   • in source docs - generate api docs automataically using apidocs.py (Issue Generate API docs from Cython sources #204)
   • update wiki pages and move to docs/
   • new docs: tutorial, migration-guide
9. Python 3 support - update to latest Cython
10. CI builds with Travis and AppVeyor

Drafts:

• API.draft - https://github.com/cztomczak/pycef/blob/master/API.draft
• TODO - https://github.com/cztomczak/pycef/blob/master/TODO

Some of the upcoming changes:

• Remove big binary files from git history - Issue Remove big binary files from git history #103 - after this is done users will have to rebase any branches they have created
• Create an archive 31.2 release and a "cefpython31" branch - Issue Create "chrome31" branch and an archive 31.2 release before updating to a newer CEF #172.
• Documentation improvements - add TOC, remove CEF 1 API, restructure - Issue Documentation improvements (TOC, remove depracted CEF 1, restructure) #150
• Remove CEF 1 code path - Issue Remove CEF 1 code path  #106
• Update to Chrome 47 - Issue Update to Chrome 47 #191
• Update to the latest Cython 0.23.4 - Issue Update Cython to latest #110
• Chrome 47 RC for Py2/3 - Issue Chrome 51 RC for Py2 and Py3 #213
• Use cmakefiles and ninja on Windows - Issue Create tools/build.py. Automate building C++ libs and subprocess and make it cross-platform. Windows platform already supported. #210
• Create examples/ directory and move all examples there - Issue Create examples/ root dir - simplify and unify examples to work on all platforms #207
• Start adding API documentation in Cython sources (.pyx) - Issue Generate API docs from Cython sources #204
• ApplicationSettings and BrowserSettings as classes - Issue ApplicationSettings and BrowserSettings as classes #205
• Make docs searchable using repository's code search - Issue Make documentation searchable using GitHub's code search #211

Maybe if time allows:

• Unit tests support - Issue Unit tests support. Partially done. #59
• CI builds - Issue CI builds - Travis (Mac, Linux), AppVeyor (Win) and Shippable (ARM) #202

With the upcoming Python 3.4 and Python 3.5 releases there will lots of python flavors and OS'es to test it all, we need unit tests support to automate testing.

Other todo:

• NetworkError.md - make this a class
• VirtualKey.md - make this a class and also See Issue Create unified key mappings for all platforms #168 "Create unified key mappings for all platforms", cefpython.keyboard module
• Use @deprecated tag in docstring in Cython sources, meaning that this function is depracated and kept only for backwards compatibility reasons - do not include such function when generating documentation.
• Update copyright notices: (c) 2012 is sufficient, you do not need to include subsequent years. Include only year in which the file was created.

CEF update related issues:

• Allow command line switches to contain "-" or "--" and update documentation (this probably was fixed to be allowed in latest CEF)
• changes in idrissha fork - Issue idrissha's fork - modifications, new features and changes in upstream cefclient #216

Restructure files and directories:

• cef1/ - remove
• cef3/* - go to /
• cef3/wx-subpackage/ - rename to wx
• cef3/client_handler/ - move to root / where .pyx files are - so that .cpp .h .pyx and .pxd files are all in the same place (no need to run through three different directories to modify download handler behavior).
  • with the exception of handlers - move these (.cpp, .h, .pyx, .pxd) to /handlers/ but other non-handler .cpp and .h files to root / directory
  • .vcproj - remove (Issue Create tools/build.py. Automate building C++ libs and subprocess and make it cross-platform. Windows platform already supported. #210 - Use cmakefiles and ninja on Windows)
  • client_handler.cpp - seperate into multiple handlers (lifespan_handler.cpp,
    display_handler.cpp, etc)
  • OpenInExternalBrowser() - move to util_linux.cpp
  • dpi_aware.cpp - rename to dpi_aware_win.cpp
• linux/binaries_32bit/ (also mac/win) - rename to binaries32/ and move to /build/binaries32/ (do not create dir in git repo, same for build/)
• linux/setup/lib_32bit/ (also mac/win) - rename to lib32/ and move to /build/lib32/ (do not create dir in git repo), remove README in windows/setup/lib_32bit/
• cef3/subprocess/ - move to / . Some handlers may be shared in both browser and renderer process.
  • cefpython_app.cpp - rename to cef_app_impl.cpp, also extract code into several files:
    • browser_process_handler.cpp
    • render_process_handler.cpp
    • javascript_bindings.cpp
• cef3/DebugLog.h and LOG_DEBUG.h - combine into "debug.h" and move to /
• cef3/util.h - move to / and rename to base.h or common.h
• cef3/windows/stdint.h - rename to stdint_win.h and move to /
• var/ - get rid of
• tests/test-launch.sh - rename to issue131-stress-test.sh and move to /tools/
• cpp_utils/* - move to /
• linux/setup/, win/setup/, mac/setup/
  • move these to one cython/ directory. fix_pyxfiles.py - duplicate file in linux/ windows/ and mac/ directories
  • cefpython.h - get rid of. It's not needed, but when missing setup.py must be called at least twice to build properly, once to generate .h file, another one to make final build - refactor build.py so that it will call itself twice automatically when required.
• move ctags.bat content to the InternalDevelopment wiki page

Restructure Cython pyx/pxd files:

• Take a quick read through Cython's Documentation, Wiki and Issue Tracker for better understand of Cython:
  • http://docs.cython.org/index.html
  • https://github.com/cython/cython/wiki
  • http://trac.cython.org/report/1
  • Also take a look at Kivy's Cython code: https://github.com/kivy/kivy
• get rid of the "combining all pyx files into one big pyx" hack - use pyx files as modules.
  • See setup's cythonize(["*.pyx"]): https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
• get rid of "imports.pyx" - do cimport in appropriate pyx modules
• other non CEF include/ c imports move from cython_includes/ to extern/
• move .pyx handlers (display_handler.pyx, v8context_handler.pyx) to the same directory in which c++ implementations reside, to / so that display_handler.cpp and display_handler.pyx are next to each other (easy to find and edit both files).
  • also applies to app.pyx
• extract settings.pyx to application_settings.pyx and browser_settings.pyx
• request_cef.pyx - Request.Flags dict convert to RequestFlags class (auto completion)
• move any constants to their classes equivalent (but keep old constants backwards compatible) - Issue Create translator.py tool to provide full CEF API coverage in an automated way  #214 "Create "translator.py" tool that automatically generates Cython source code based on the CEF header files".
• Issue Expose handlers as classes/interfaces: ClientHandler, DisplayHandler, LifespanHandler, etc. #215 "Expose handlers as classes/interfaces: ClientHandler, DisplayHandler, LifespanHandler, etc."
• rename process_message_utils.pyx to "ipc_data_conversions.pyx"
• move cef_*.pxd files from "cython_includes/" to upper directory and rename these files to match the equivalent .pyx file (eg. cef_browser.pxd -> browser.pyx + browser.pxd).
  • for handlers move both .pxd and .pyx file to handlers/ (along with .cpp, .h)
• move GetModuleDirectory() from utils.pyx to helpers.pyx (cefpython.helpers module), but keep it also BC available in the cefpython module namespace
• create deprecated.pyx - keep BC deprecated imports there, eg. import GetModuleDirectory from helpers
• get rid of *_static.pxd files. For example remove cef_browser_static.pxd - rename CreateBrowserSync to CefBrowserHost_CreateBrowserSync using this Cython syntax:
  cdef CefRefPtr[CefProcessMessage] CefProcessMessage_Create \ "CefProcessMessage::Create"(const CefString& name)
• move Debug() from utils.pyx to debug.pyx - to have it next to debug.h (+ combined LOG_DEBUG.h)
• utils.pyx: make a distinction of functions that are exposed to the cefpython module and those that are used internally in cython
  • IsFunctionOrMethod(), IsString() used only internally, should be cdef. Move to types_utils.pyx?
  • Debug() used only internally, cdef
  • IsThread() is exposed, move to thread.pyx module and rename to GetThreadId (multiple if/elseif with calls to CefCurrentlyOn for all threads - else should return Thread.Unknown). Move TID_UI constants from utils.pyx to cefpython.thread as class Thread (Thread.Ui, Thread.Io, Thread.Renderer, Thread.Db). Keep IsThread() BC in deprecated.pyx.
  • GetSystemError() - not used, remove or move to win32.pyx (and rename to GetLastError - resolve name conflict by not cimporting * from win32.pxd)
• windows.pxd - rename to winapi.pxd (to make it more obvious that these are WIN API imports)
• cefpython.helpers - helper functions not directly related to CEF
  • GetModuleDirectory() from utils.pyx
  • GetApplicationPath() from examples (wxpython.py and others)
  • ExceptHook() from examples (wxpython.py and others)
• cefpython.utils - cefpython utilities (not available in upstream CEF)
  • GetNavigateUrl() from utils.pyx
  • GetAppSetting() - move BC from cefpython
  • GetBrowserByWindowHandle() move BC from cefpython
  • GetCommandLineSwitch() move BC from cefpython
• cefpython.handlers
  • GetGlobalClientCallback() move BC from cefpython, rename to GetGlobalCallback()
  • SetGlobalClientCallback() move BC from cefpython, rename to SetGlobalCallback()
• cefpython.keyboard - unified key mappings for all platforms, Issue Create unified key mappings for all platforms #168
  • IsKeyModifier() move BC from cefpython to cefpython.keyboard
• cefpython.task module
  • PostTask() move BC from cefpython to cefpython.task
• cefpython.gtk
  • move BC gtk_plug_new() from cefpython.WindowUtils
  • move BC gtk_widget_show() from cefpython.WindowUtils
• cefpython.x11 - InstallErrorHandlers() to set default error handlers using XSetErrorHandler and XSetIOErrorHandler
• cefpython.WindowUtils - GetParentHandle and IsWindowHandle return 0/True always on Mac/Linux. Other functions are Win-only. Update documentation to acknowledge that.

Deprecated API:

• Should deprecated API be enabled by default? Or should we disable it and allow for from cefpython.deprecated import *? If deprecated is enabled by default auto completion will still show deprecated functions. Maybe cefpython.py API dummy script (Issue A dummy Python script with complete CEF Python API #58) would overwrite auto completion in IDE?
• Should we display a warning when using a deprecated function? With information that this function was moved to cefpython.utils or cefpython.helpers etc.

Tools in the /tools/ directory:

• linux/installer/, mac/installer/, windows/installer/ refactoring
  • move these to /tools/
  • /tools/make_distrib.py - call scripts in deb_pkg/ innosetup/ setup/
• /tools/setup/
  • make-setup.py (win, mac, linux) -> setup/make.py
• /tools/innosetup/
  • windows/make-installer.py -> move and rename to innosetup/make.py
• /tools/deb_pkg/ 0 linux/find-deps.py -> find_deps.py, linux/make-deb.py -> make.py
• linux/, mac/, windows/ scripts:
  • compile.py (mac/linux), compile.bat, compile.cmd, rebuild.sh, wxpython.sh, mac32.sh, mac64.sh, wxpython.bat - move these to /tools/:
    • /tools/build.py - rebuild only if needed, unless "force" flag is passed
    • /tools/run.py (run.py wx, run.py wx all, run.py gtk, run.py qt) - 1. rebuild only if sources changed 2. install package if rebuilt, 3. run example [all - run all examples, "wx all" - run all wx examples].
• /tools/unittests.py - rebuild if needed, install package if rebuilt, run unit tests (Issue Unit tests support. Partially done. #59 "Unit tests support")
• /tools/test_all.py - test everything possible:
  • rebuild if needed
  • build packages
  • test all packages, for each package:
    1. install package
    2. run unit tests
    3. run all examples (that work on that platform - each example should have embedded in comments one of these tags: @linux @windows @mac - meaning it works on such platform)
    4. uninstall package
• /tools/apidocs.py - generate API markdown pages in /docs/ directory (also use doctoc?):
  • handlers/
  • objects/
  • settings/
  • modules/
  • API.md
  • eacho of auto generated .md files should have an html comment at top saying it was auto-generated
• /tools/automate.py
  1. download CEF binaries and symbols
     • read CEF branch / revision from _VERSION.pyx (cef_version.h file is auto-generated and available only in distribs, not available in cef git sources - we shouldn't use it as it is tied to a platform specific distribution)
     • download binary from GitHub Releases which will be tagged "upstream-cef47", which should include Release Symbols as well
     • binaries should be named the same as on cefbuilds.com eg. "cef_binary_3.2526.1366.g8617e7c_windows32.7z"
  2. build
  3. optionally install, run unit tests and run (if flag "run").