Cython bindings oss build

[gmarzot]

see attached file for bootstrapping python/cython venv into build environment

moved interface source files to not collide with cython gen names:
iobuf.cpp/h => iobuf_intf.cpp/h
executor.cpp/h => executor_intf.cpp/h
include paths for all cython generated .h _api.h <folly/...> and i left all the source references as <folly/python/...>
updated CMake ignore lists
small tweak to support python 3.13 Py_IsFinalizing()
rewrite setup.py - build all packages (folly, folly.iobuf, folly.executor, folly.fiber_manager, folly.build_mode)

several manifest dependency issues:
xz url redirects to sourceforge(?) and fails download (how is this working for anyone?)
ninja rather old version not compatible with modern python 3.13
libiberty was not supporting -fPIC needed to link to python bindings (hard-coded this change - could be improved - please review)

various changes to eliminate warnings from cython 3.0.11-1
I suggest eliminating all python support for versions prior to 3.0 and likewise requiring >=3.0 for cython would be reasonable.

I am building with PYTHON_EXTENSIONS ON like this:

./build/fbcode_builder/getdeps.py build --extra-cmake-defines '{ "CMAKE_CXX_STANDARD": "20", "CMAKE_CXX_FLAGS": "-std=c++20 -fPIC", "PYTHON_EXTENSIONS": "ON", "BUILD_SHARED_LIBS": "ON", "BOOST_LINK_STATIC": "ON" }' --extra-b2-args "cxxflags=-fPIC" --extra-b2-args "cflags=-fPIC" --no-build-cache --no-tests --scratch-path /tmp/folly | tee "/tmp/folly_build-$(date +%Y%m%d_%H%M).log"

simple test like this:

export PYTHONPATH=<output from build> 
export LD_LIBRARY_PATH=<output from build> #macos you set DYLD_LIBRARY_PATH instead of LD_LIBRARY_PATH
python3 -c "import folly; print('folly import successful')"
python3 -c "from folly import executor; print('folly executor import successful')"
python3 -c "from folly import iobuf; print('folly iobuf import successful')"
python3 -c "from folly import fiber_manager; print('folly fiber_manager import successful')"

once the cython build above completes and installs, the test directory can be built from folly/python/test like this:

export PYTHONPATH=<output from build> 
export LD_LIBRARY_PATH=<output from build> #macos you set DYLD_LIBRARY_PATH instead of LD_LIBRARY_PATH
python3 setup.py build_ext --inplace --include-dirs="${GETDEPS_INSTALL_DIR}/folly/lib/python3.13/site-packages:$(pwd)/../../../" --force

the tests may then be run from folly/python like this:

pytest -n auto -v test

MacOS Specific Setup:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)
echo 'eval "$(/usr/local/bin/brew shellenv)"' >> ~/.zprofile
brew install git
# mac-dev-setup.sh may be overkill
curl https://raw.githubusercontent.com/thomaspoignant/mac-dev-setup/master/src/mac-dev-setup.sh | bash
xcode-select --install
brew install --cask autopkgr
brew install autoconf
brew install autopoint
brew install po4a
brew install gettext

[gmarzot]

relates to Issue #2358
relates to Issue #1703

[gmarzot]

for reference I am using the following script to bootstrap Python and Cython into the folly build environment

ECHO=$([[ "$SHELL" == *"zsh"* ]] && echo "echo" || echo "echo -e")

$ECHO "${COLOR_GREEN}Setting up Python test environment ${COLOR_OFF}"
PYTHON_VERSION="3.13"   # Latest Python version to install via uv
CYTHON_VERSION="3.0.11" # Target Cython release (3.1.0a1 crashes - use 3.1.0 when released)

PLATFORM="$(uname -s)"
case "${PLATFORM}" in
    Linux*)     PLATFORM=Linux;;
    Darwin*)    PLATFORM=Mac;;
    *)          PLATFORM="UNKNOWN:${PLATFORM}"
esac
$ECHO "${COLOR_GREEN}Detected platform: $PLATFORM ${COLOR_OFF}"

# Ensure python3 and pip3 are installed
if [ "${PLATFORM}" = "Linux" ]; then
    if [ "${EUID}" -ne 0 ]; then
	SUDO=sudo
    fi
    ${SUDO} apt-get install -y python3 python3-pip python3-venv clang
elif [ "${PLATFORM}" = "Mac" ]; then
    brew install python3
else
    $ECHO "${COLOR_RED}[ ERROR ] Unrecognised platform: ${PLATFORM}${COLOR_OFF}"
    exit 1
fi 

VENV_DIR=".venv"
UV_PIP_INSTALL="uv pip install --no-config --upgrade"
export UV_PYTHON_INSTALL_DIR="${VENV_DIR}"

if [ ! -d "${VENV_DIR}" ]; then
    [ -f ".python-version" ] && rm -f ".python-version"
    python3 -m venv --prompt "pip" "${VENV_DIR}"
    source "${VENV_DIR}/bin/activate"
    python3 -m pip install --upgrade pip
    python3 -m pip install --upgrade uv
    uv venv --prompt "uv" \
        --seed --relocatable \
        --allow-existing --no-config \
        --python-preference only-managed \
        --python "${PYTHON_VERSION}" \
        "${VENV_DIR}"
fi

source ./.venv/bin/activate
if ! command -v "uv" >/dev/null 2>&1 ; then
    $ECHO "${COLOR_RED}[ ERROR ] Failed to install uv... ${COLOR_OFF}"
    exit -1
fi

# Ensure uv is upgraded to latest
${UV_PIP_INSTALL} uv

# Install desired python verion using uv
uv python install --no-config "${PYTHON_VERSION}"
uv python pin --no-config "${PYTHON_VERSION}" 

# Install required packages
${UV_PIP_INSTALL} build wheel setuptools
#${UV_PIP_INSTALL} "git+https://github.com/cython/cython.git@${CYTHON_VERSION}"
${UV_PIP_INSTALL} "cython==${CYTHON_VERSION}"
${UV_PIP_INSTALL} pytest pytest-asyncio pytest-cov 
${UV_PIP_INSTALL} aioquic

cd "${VENV_DIR}"
PYTHON_DIR=$(python3 -c 'import sysconfig; print(sysconfig.get_config_var("prefix").split("/")[-1])')
if [ -d "include" ] && [ ! -L "include" ]; then
    rm -rf "include"
fi
ln -sf "${PYTHON_DIR}/include" "include"
cd ..
deactivate

$ECHO "${COLOR_GREEN}Python test environment is set up ${COLOR_OFF}"

[gmarzot]

folly_cython_macos_tests.txt

MacOS builds and tests successfully

[facebook-github-bot]

@Orvid has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[jmswen]

@gmarzot, I discovered through @Orvid that you and I have been doing similar work recently to revive the build for Python support in folly. (For additional context, I'm also working on reviving support for Python in the fbthrift repo as well.) My similar PR is in #2370. Can you let me know if that PR works for you? I noticed that your PR also includes support for newer extensions – mine only tries to fix up support for folly.iobuf and folly.executor at the moment, but it would be easy to extend the changes.

[gmarzot]

hi jmswen, can you supply the specific build command you are running to execute the build (python cython version, platform etc.)? Were you able to run any tests against the packages produced? attaching output like this to PR is generally good. You will see in my PR these details and a successful run of the unit tests in the folly/python/test directory. At first glance I see my PR as a superset of these changes. Though I do see some things at the cmake level that I could merge in. If we can agree my PR covers all your needs, then we can recommend to @Orvid King ***@***.***> how to proceed. thanks for contacting me and best regards, Giovanni

…

On Thu, Jan 16, 2025 at 10:19 PM jmswen ***@***.***> wrote: @gmarzot <https://github.com/gmarzot>, I discovered through @Orvid <https://github.com/Orvid> that you and I have been doing similar work recently to revive the build for Python support in folly. (For additional context, I'm also working on reviving support for Python in the fbthrift repo as well.) My similar PR is in #2370 <#2370>. Can you let me know if that PR works for you? I noticed that your PR also includes support for newer extensions – mine only tries to fix up support for folly.iobuf and folly.executor at the moment, but it would be easy to extend the changes. — Reply to this email directly, view it on GitHub <#2361 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7T3IEYX4O5A43BEXNSC7D2LBZE3AVCNFSM6AAAAABU4N5YAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOJXGM2TCNBYGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jmswen]

@gmarzot It seems the build/test commands were stripped from the commit info when exporting to GitHub from Meta's internal Phabricator. The commands I'm using (which you'll likely need to adapt slightly) are:

# Create a new venv, assume cwd is $PROJECT_DIR
python3.12 -m venv .venv
source .venv/bin/activate
pip install setuptools cython

# Build folly with Python extensions
cd ~/folly
CPLUS_INCLUDE_PATH="$CPLUS_INCLUDE_PATH:/usr/local/fbcode/platform010/include/python3.12/" ./build/fbcode_builder/getdeps.py build folly --extra-cmake-defines '{"PYTHON_EXTENSIONS": "ON", "BUILD_SHARED_LIBS": "ON", "CMAKE_POSITION_INDEPENDENT_CODE": "ON", "CMAKE_CXX_STANDARD": "17", "PYTHON_INSTALL_DIR": "$PROJECT_DIR/.venv"}' --extra-b2-args "cxxflags=-fPIC" --extra-b2-args "cflags=-fPIC" --verbose

# Run some basic tests, first requires setting `LD_LIBRARY_PATH` (similar to what you did in your tests)
python -c 'from folly import iobuf; a = iobuf.IOBuf(b"abcd"); print(len(a))'

As I mentioned, I have also been working on reviving the fbthrift Python support, and I have verified that an fbthrift Python service works with my folly changes in #2370.

As you alluded to, your PR makes more changes than mine. Given that it will likely take some time to review all those changes, I think the best path forward is to land my PR and rebase the changes related to folly/python/setup.py on top of that (many of the other changes in your PR are orthogonal and would require minimal rebase effort).

[gmarzot]

As you alluded to, your PR makes more changes than mine. Given that it will likely take some time to review all those changes, I think the best path forward is to land my PR and rebase the changes related to folly/python/setup.py on top of that (many of the other changes in your PR are orthogonal and would require minimal rebase effort).

@jmswen yes i guess not 100% orthogonal but we could proceed as you suggest. @Orvid not sure what input you need but i am fine with the suggestion above. @jmswen thank you for your work and contribution and i will check out the fbthrift stuff (not familiar). in the meantime i will merge you pr and see how it plays.. i am curious why you have c++17 vs c++20? and i am trying to see how you were able to seemingly skip some of the lib dependencies i ran into.. did you try using cython 3.1 dev? the iobuf pyx crashes compile for me.