Python the Language (in case of)

Python the Language (in case of)

2016-08-21. Category & Tags: Python, Environment, Virtualenv, Py, Py3, Lang, Language

PRE-INSTALL: CONDA VS. PIP #

Pip requires apt. Thus, pip+apt. Pip chose to live together with apt (or other package manager) as apt is better at managing packages, especially when related to system. Some pip packages requires system (apt) to cooperate, such as uwsgi, graphviz, ffmpeg, gcc, python3-dev.

For data / AI engineering, conda > pip. Conda has a good system and community to maintain AI/ML related packages, but for tasks beyond AI/ML, pip is sometimes required. It even breaks during env migration and has to be fixed by pip. Thus, conda+pip+apt, which is more complex than pip+apt for engineering other than AI/ML.

ref: zhihu (bak)

MINICONDA/ANACONDA (YML) VIRTUAL ENV #

Anaconda 安装包可以到 https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/ 下载。

If possible, (mini)conda (and its virtual env) is suggested for easy installation of binary libs, however, a few libs may lack specific binary versions in anaconda or conda-forge channels, such as scipy==1.2.2.

install on win (tested on win 10) #

Py3 64bit, or: wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe

quiet mode:

start /wait "" Miniconda3-latest-Windows-x86_64.exe /InstallationType=JustMe /RegisterPython=0 /S /D=%UserProfile%\Miniconda3

install on nix (tested on ubuntu 20, quiet mode) #

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \
bash ~/miniconda.sh -b -p $HOME/miniconda && \
eval "$(~/miniconda/bin/conda shell.bash hook)" && \
$HOME/miniconda/bin/conda init

init .profile in windows/pycharm or linux #

# for windows
conda init powershell # then re-open ps

# or for nix
conda init bash # then re-open bash

init env #

install py & pip: #

Python is also considered as a ‘package’ like other packages.
The py version can be specified when creating an env: conda create -p ./condaEnvWin_Nix python==3.6 or install like a package after activating an env: conda install python==3.6 pip.
Tip: the suggested py version is 2 minor versions behind current release due to slower release of packages. E.g., if the current official release is 3.10.2, then 3.8.x is suggested.

install using pip in conda env.: #

conda install pip
which -a pip
unset PYTHONPATH
pip install ...

ref: possible issues and solution

install specific package/lib version from source: #

Install pip inside an env and then use this pip (tip: which pip) to install required packages when channels lack it.

suggested mirrors & channels: #

mirror: tsinghua-Tuna for ~/.condarc

conda config <--env> --add channels new_channel # --add == --prepend, add to list top with highest priority
conda config <--env> --append channels new_channel

Tip 1: --env configurates only the current env.
Tip 2: Strict channel priority can dramatically speed up conda operations and also reduce package incompatibility problems, but may break old env files.
Tip 3: to change Pycharm channels, modify C:\Users\<win-username>\.condarc as given by conda config --show-sources.
Tip 4: for PyTorch AI, you may want the channel: fastchan.

WARN: mixing channels / using multiple channels may cause incompatible issues/conflicts [ref].

Useful cmd/config: channel_priority: true in ‘.condarc’ or conda config --set channel_priority true;
conda config --describe channel_priority

ref: Managing channels, v4.11 doc

usage #

suggested usage: #

conda create -p  condaEnvWin_or_Nix && \
conda activate ./condaEnvWin_or_Nix && \
conda deactivate
conda config --add channels conda-forge

which will put everything in the new folder of current folder (e.g. current project folder).(-p is --prefix)

Tip 1: differnt envs can have the same folder name (as the full path is different).
Tip 2: full path can be used to activate an env when pwd is in another place.

Also suggested:

conda activate ./condaEnvWin_or_Nix && \
conda install -y -c conda-forge notebook nb_conda ipykernel ipywidgets jupyterthemes && \
jupyter notebook --generate-config && \
jupyter notebook

DIR_NAME=${PWD##*/} && ipython kernel install --name "$DIR_NAME" --user
# or:
DIR_NAME=${PWD##*/} && python3 -m ipykernel install --user --name=DIR_NAME

Note:
nb_conda: add jupyter kernel.
generate-config: generate default config file (so we can edit configs).

another way (default folder): #

To put into the default ‘$HOME/.(mini)conda/envs/’ folder: conda create condaEnvWin_or_Nix && conda activate condaEnvWin_or_Nix.

create from a file: #

conda [env] create -f environment.yml or conda [env] create -f requirements.txt.
(-f is --file, plz note this shortcut is NOT available with conda install)

export to a file: #

Suggested: conda env export --from-history > environment.yml that includes packages which are explicitly asked for by the user (dependencies not included). This environment file works across platforms.
Or, conda env export > environment.yml that includes all installed packages (for the same platform).

other tips #

sys env variables
check env changes/history and restore an env
List all envs: conda info --envs or conda env list.
List all channels: conda config --show channels.
Clean cache: conda clean -a (== --all).
-y : To ignore all questions when creating an env etc.
WARN: if one env is moved manually, the file ‘~/.conda/environments.txt’ should be modified manually, too.
Suggested alias:

alias conc='conda create -y -p ./condaEnvNix'
alias cona='conda activate ./condaEnvNix'
alias cond='conda deactivate'

FAQ:
Problem: “failed with initial frozen solve” or “Found conflicts! Looking for incompatible packages.”
Reason: technical debt of conda (so, bug?).
Solution: conda config --set channel_priority flexible or conda update --all --yes or conda list --revisions; conda install --revision <rev_num> or pip (which should almost always work). A good practice might be always using pip to install if possible and treating conda as mainly a venv manager and a spare pkg manager.

INSTALL PY3 #

win #

Install py2 first! Install py3 later.
Download and install py3 from the official website.
During installation, make sure to “add to system PATH”.
If it is only in user’s “Path”, we need to also add to system “PATH” (both the python root folder and the script subfolder), and make sure it is before py2’s folders.

ubuntu #

Plz use conda venv/pyenv below.
(Ubuntu 16.04 has both py2 & py3 but default to py2.)

INSTALL PIP IN UBUNTU 16 #

Suggested: use conda venv.

sudo apt update && sudo apt -y upgrade && \
sudo apt install python-pip python3-pip && \
pip -V; pip3 -V

# use a local index/source instead of https://pypi.python.org/simple/
pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple/ pip -U
pip3 config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple/

Alternative indexes/sources [ref]:

清华大学 https://pypi.tuna.tsinghua.edu.cn/simple/
阿里云 http://mirrors.aliyun.com/pypi/simple/
豆瓣(douban) http://pypi.douban.com/simple/
中国科学技术大学 http://pypi.mirrors.ustc.edu.cn/simple/ (302 redirected to mirrors.tuna.tsinghua.edu.cn)

FREQUENT PIP CMD #

pip install --prefer-binary package_name
pip uninstall package_name
pip search package_name
pip freeze
pip --help

JUPYTER: see: ./jupyter

PYENV #

Suggested: use conda venv.

installing pyenv fr. source #

WARN: Before installing any specific py version:
sudo apt install -y python3-pip curl git python3-dev libbz2-dev libssl-dev libsqlite3-dev libreadline-dev lib32readline-dev libffi-dev build-essential zlib1g-dev.
(see the common error below for the reason)

Run curl [ref]:
(manually do it if lacking curl)

Note: every user needs to run individually:

curl https://pyenv.run | bash

# for cn users:
set -e && curl -s -S -L https://tinyurl.com/pyenv-gitee | bash
# or:
set -e && curl -s -S -L https://clck.ru/UBQPr | bash

Update .bashrc:

# pyenv
if [ -d "$HOME/.pyenv" ]; then
  export PYENV_ROOT="$HOME/.pyenv"
  export PATH="$PYENV_ROOT/bin:$PATH"
  eval "$(pyenv init -)"
  eval "$(pyenv virtualenv-init -)"
fi

Then:

source ~/.bash_profile

Install python in pyenv:

pyenv install 2.7.18

# for cn users (10x speed):
v=2.7.18;wget https://npm.taobao.org/mirrors/python/$v/Python-$v.tar.xz -P ~/.pyenv/cache/;pyenv install $v

common error (Ubuntu20) #

QA: a common error when installing a python version using pyenv install ...:

WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib
ERROR: The Python ssl extension was not compiled. Missing the OpenSSL lib

Solution:
See above “Before installing any specific py version”. Note: may need to search by: apt-cache search readline |grep dev

[ref libffi requies re-install of py3]

check py version #

Tip: to find py version:

# in terminal:
python --version
# in py/jupyter:
import sys
sys.version_info

print(sys.executable) # https://stackoverflow.com/a/46902466
print(sys.version)
print(sys.version_info)

PIPENV #

Suggested: use conda venv.

A state-of-the-art combined tool chain.

install pipenv #

Tip: better to have pyenv installed also, so any python version can be automatically installed by pipenv.

sudo apt install -y python3-pip python3-setuptools && \
python3 -m pip install pipenv # Note: plz use --user when installing pipenv for non-root users using sudo

cat >>~/.bash_profile <<EOD
export PATH="\$HOME/.local/bin:\$PATH"
EOD
source ~/.bash_profile

create env #

mkdir sunnyProj; cd sunnyProj
pipenv --python 3.6.7 # will automatically install the specified python in env, if pyenv is installed.

(note: if got error, try manually install a python version by pyenv install x.x.x w/o ‘rc’, then pipenv ... again.)

activate env #

pipenv shell

how it works #

A pip-environment is based on one folder.
Actually, the Pipfile in the folder.
When creating a new environment, it will use the folder’s name as the environment’s name, and will also search for the environment using the folder’s anme when trying to activating by pipenv shell.
When activating one environment, a specific Python’s installation path and Python lib path will be added to PATH and have higher priority than system one.
“Pipenv” uses “virtualenvs” as the backend (?).

Ref:
Install and kick-start, (originally from phoikoi.io)

pipenv (bak) (we can use pipenv --python directly as it will install the related python)

list of packages/libraries #

Bash:

pip freeze # only the ones installed by `pip install xxx`
pip freeze --all
pip list --format freeze
pipreqs /<your_project_root_path>/ # get requirements by checking `import`s in the project. ref: https://towardsdatascience.com/goodbye-pip-freeze-welcome-pipreqs-258d2e7a5a62 bak: tinyurl/4anrdnca

Python:

import pip
installed_packages = pip.get_installed_distributions()
installed_packages_list = sorted(["%s==%s" % (i.key, i.version)
     for i in installed_packages])
print(installed_packages_list)

# OR (one-line)
sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()])

Ref: stackoverflow

pipenv + jupyter kernels #

see /jupyter

VIRTUAL ENV #

OBS: dated, see pipenv above instead.

There are 2 tools (virtualenv and virtualenvwrapper) can be used. There is also one blog introducing bash alias way to use virtualenv: ref ( evernote backup ).
We will use the wrapper one (cuz we prefer new stuff, right?).
If we need to put the env into a folder other than the user’s home, we can use the old virtualenv package:
virtualenv [-p /usr/bin/python2.6] <path/to/new/virtualenv/>

// ## install

Win: pip install --prefer-binary virtualenvwrapper-win
Nix: pip install --prefer-binary virtualenvwrapper (this also works for windows with cygwin)

// ## make new env

mkvirtualenv <env name>
If we want to use another python intepretor, we can use the optional -p (path) parameter:
mkvirtualenv -p D:\Python27\python.exe py27_rest_demo
TIP: After running this command, we will automatically enter the virtual environment.
The name is showing at the beginning of terminal/cmd prompt, surrended by ( ):

OBS: newly created env does NOT have globally installed packages.

// ## using previously created env

workon <env name>

// ## packages

When we are in an env, we can use pip to install packages for this env.

// ## quit current env

deactivate

// ## test package version

Take Django as an example:

python

import django
print(django.get_version())

// ## cmd format

Even with -p parameter, we need to run the .py files with virtual env’s python.exe, e.g.:
C:\Users\<user name>\Envs\py27_rest_demo\Scripts\python.exe manage.py runserver
Otherwise, system’s global intepretor will take effect.
Easier Way
We can use D:\Python27>mklink python27.exe python.exe to make it possible to run server by: python27.exe manage.py runserver. However, direct copying python.exe and renaming the file does NOT work.

POETRY (alt. to pipenv) #

Suggested: use conda venv.

install #

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
poetry config virtualenvs.in-project true

add to project config file pyproject.toml:

[[tool.poetry.source]]
name = "pypi-tuna-simple"
url = "https://pypi.tuna.tsinghua.edu.cn/simple/"
default = true

usage #

// Run pyenv local 3.7 once to create a file .python-version to remember desired py version.

pyenv shell 3.6.9
poetry shell

note #

advantages: good dependency resolver among libraries; also support jupyter notebook/lab.
disadvantages: poor dependency resolver when python version does not match (need to manually specify ); cannot automatically call pyenv, manual work needed.

PARALLEL #

joblib (see my own ipynb)
numba (bak)

PERFORMANCE #

Simple seconds: time()

import time
start_time = time.time()
# do something here
end_time = time.time()
print(end_time - start_time, 'seconds.')

Alternatives: perf_counter or process_time
ref stackoverflow.

Python in Browser #

Use Py instead of js (javascript) by pyodide.
Try directly here.
Maybe also Juliea & R later.

MORE #

automated lazy-loading of common libs: pip install --prefer-binary pyforest
不可不知的 Python 陷阱
 Kite: Python 代码补全利器

HANDY LIBS #

python -m http.server [8000] # default port 8000, seems no need to install
python -m SimpleHTTPServer  [8000] # default port 8000

PACK PY PROJECTS (django) AS .EXE FOR WINDOWS #

winpython #

pyinstaller tips #

WARNING !!!

Check the supported python version and django versions even before coding.
The Python distribution version from windows store is lacking something, so DO NOT USE !!!
(a good alt. is the .org official one or the winpython dist.)

Problem: “RuntimeError: Script ‘runserver’ does not exist”
Reason: // possible compatibility issue due to newer django
Solution: manage runserver --noreload [ref]

Problem: ModuleNotFoundError ...
Reason 1: using custom app verbose_name in my_site/my_app/apps.py
Solution 1: not use :(
Solution 2: see the csdn ref below, use --hidden-import and/or hook during compiling (more problems: need to import xxx.urls etc.)
Solution 3: import them in manage.py before compiling instead of using --hidden-import etc. flags.
Reason: other.
Solution: csdn

Problem: ImportError: cannot import name _elementpath
Reason: using python-docx (and some deeper reasons)
Solution: in manage.py: from lxml import _elementpath

Problem: template not found.
Reason: pyinstaller cannot identify & copy html templates & static folders.
Solution: manual copy. (a better one???)

Problem: .pyc is not safe.
Reason: every single line can be de-compiled easily from .pyc files, including comments.
Solution: to .pyd by from Cython.Build import cythonize ...

Problem: wanna get .py from pyinstaller generated .exe.
Solution: python pyinstxtractor.py manage.exe; pip install --prefer-binary uncompyle6; uncompyle6 model.pyc ref cn

Problem: “…ImproperlyConfigured: Requested setting DJANGO_TABLES2_TEMPLATE, but settings are not configured.”
Reason: lacks the config.
Solution: DJANGO_TABLES2_TEMPLATE = "django_tables2/bootstrap4.html" in settings.py. [refs: doc 1.11.2 & blog] and SET DJANGO_SETTINGS_MODULE=<my_site_name>.settings (like export in nix) before running [ref1, [ref2]].

Problem: static file dir config not working when DEBUG=False.
Reason: …
Solution: …

Problem: pyinstaller compiled exe files detected as virus.
Reason & solution: https://stackoverflow.com/a/52054580
Solution NOT working !

For meanings of <my_site_name>.spec, see ref cn (evernote bak)

some more tips: [full process in linux, csdn 2020]; [hidden imports, csdn 2020]

some more pitfalls (mentioning similar ones but different solutions):

Problem: MIME type not matched, so js not loaded/executed.
Reason: pyCharm send MIME correctly, but pyinstaller send .js files’ MIME type as plain/text together with nosniff, thus the browser treat js as plain text.
Solution: in each (or relevant) site folder’s settings.py:

import mimetypes
mimetypes.add_type("application/javascript", ".js", True)

nt solutions):

import mimetypes
mimetypes.add_type("application/javascript", ".js", True)

Problem: “Could not find a version that satisfies the requirement setuptools (from versions: none)”
If you want the build to “see” system installed packages, use pip install --no-build-isolation.