Skip to main content

Python

News​

Writing Idiomatic Python Code​

Tutorial

To output "The Zen of Python" simply run (in your interpreter):

import this

Installing multiple Python Versions​

  • source:
    • sudo add-apt-repository ppa:deadsnakes/ppa
    • sudo apt-get update
    • sudo apt-get install python3.5
    • sudo apt-get install python3.5-dev
      • It will not overwrite your existing python3.4 which is still symlinked as python3.
      • Instead, to run python3.5, run the command python3.5 (or python3.X for any other version of python).

venv using multiple Python Versions​

  • install sudo apt install python3.7-venv or sudo apt install python3.x-venv for each python version!

Uninstalling multiple Python Versions​

  • sudo apt purge python3.6 libpython3.6-minimal libpython3.6-stdlib python3.6-minimal

python packages and modules​

  • source
  • .py files are modules
  • folders (containing modules) are packages
    • importing a package essentially imports the package's __init__.py file as a module

pyenv​

  • For Python version management, e.g.
    • if you want to use multiple python versions on the same machine
    • if a project requires an older python version
  • pipenv install (see below) will automatically install a python version using pyenv, if the project requires it

Prerequisites​

sudo apt-get update; sudo apt-get install make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

Install​

curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc

exec $SHELL

pyenv update

Usage​

Remember: pyenv works by inserting a directory of shims at the front of your PATH (see Understanding Shims).

Run a program in a poetry environment​

pyenv shell 3.10.4
poetry run python3 some_python_script.py

Install Python versions​

# list available Python versions that can be installed
pyenv install -l
pyenv install 3.10.4

Switch between Python versions​

from official doc

To select a Pyenv-installed Python as the version to use, run one of the following commands:

  • pyenv shell <version> -- select just for current shell session
  • pyenv local <version> -- automatically select whenever you are in the current directory (or its subdirectories)
  • pyenv global <version> -- select globally for your user account

E.g. to select the above-mentioned newly-installed Python 3.10.4 as your preferred version to use:

pyenv global 3.10.4

Now whenever you invoke python, pip etc., an executable from the Pyenv-provided 3.10.4 installation will be run instead of the system Python.

Using "system" as a version name would reset the selection to your system-provided Python.

Shipping Python Packages​

  • as a
    • pip package hosted on PyPI
    • platform-independent zipimport binary (see eg. yt-dlp)
    • Linux standalone x64 binary
    • Linux standalone aarch64 (64-bit) binary
    • source tarball (see install packages)

pip​

  • use pipenv (see subsection below) instead of venv and pip
  • pip caches downloaded packages, so that you can resume downloads and installations by running pip install again, if e.g.
    • the installation is aborted because there was no space left on the device

Basics​

commanddescription
pip show *package*show location of package (and other package information)
pip install -hshow install options/settings
python3 -m pip install --upgrade pipupgrade pip (old pip versions cause many errors, always upgrade pip first and keep pip updated!)

PyPI​

Wikipedia

The Python Package Index, abbreviated as PyPI (/ˌpaɪpiˈaɪ/) and also known as the Cheese Shop (a reference to the Monty Python's Flying Circus sketch "Cheese Shop"), is the official third-party software repository for Python. (...) PyPI is run by the Python Software Foundation, a charity. Some package managers, including pip, use PyPI as the default source for packages and their dependencies.

As of 6 May 2024, more than 530,000 Python packages are available.

PyPI primarily hosts Python packages in the form of source archives, called "sdists", or of "wheels" that may contain binary modules from a compiled language.

Wheels vs Source Distributions​

sdist​

realpython.com

example: install the uWSGI package

The tar.gz tarball that pip retrieves is a source distribution, or sdist, rather than a wheel. In some ways, a sdist is the opposite of a wheel.

A source distribution contains source code. That includes not only Python code but also the source code of any extension modules (usually in C or C++) bundled with the package. With source distributions, extension modules are compiled on the user’s side rather than the developer’s.

Wheel​

realpython.com

example: install the chardet package

(...) Installing chardet downloads a .whl file directly from PyPI. The wheel name chardet-3.0.4-py2.py3-none-any.whl follows a specific naming convention that you’ll see later. What’s more important from the user’s perspective is that there’s no build stage when pip finds a compatible wheel on PyPI.

Wheels Make Things Go Fast​

(...) Above, you saw a comparison of an installation that fetches a prebuilt wheel and one that downloads a sdist. Wheels make the end-to-end installation of Python packages faster for two reasons:

  • All else being equal, wheels are typically smaller in size than source distributions, meaning they can move faster across a network.
  • Installing from wheels directly avoids the intermediate step of building packages off of the source distribution.

Install packages​

commanddescription
pip install -U, --upgradeUpgrade all specified packages to the newest available version. The handling of dependencies depends on the upgrade-strategy used.
pip install -q, --quietGive less output. Option is additive, and can be used up to 3 times (corresponding to WARNING, ERROR, and CRITICAL logging levels).
pip install relative_path_to_seaborn.tar.gzInstall a tar.gz source archive file ("Source tarball")
pip install absolute_path_to_seaborn.tar.gzInstall a tar.gz source archive file ("Source tarball")
pip install .pip will use setup.py to install your module. Avoid calling setup.py directly., see setup.py

setup.py​

stackoverflow

setup.py is a python file, the presence of which is an indication that the module/package you are about to install has likely been packaged and distributed with Distutils, which is the standard for distributing Python Modules.

Cache​

commanddescription
pip cache dirto get the cache directory that pip is currently configured to use
l ~/.cache/pip/a typical cache directory location
pip cache listlist all wheel files from pip’s cache
pip cache list somepackagelist all somepackage-related wheel files from pip’s cache
pip cache infoprovides an overview of the contents of pip’s cache, such as the total size and location of various parts of it
pip cache remove somepackageremoves all wheel files related to somepackage from pip’s cache. HTTP cache files are not removed at this time.
pip cache purgewill clear all files from pip’s wheel and HTTP caches.
pip install --download-cache /path/to/pip/cache matplotlibcache downloaded packages to avoid downloading them again, stackoverflow
pip install --no-cache-dirpip’s caching behaviour is disabled by passing the --no-cache-dir option.

Uninstall packages​

commanddescription
pip install pip-autoremoveutility to remove a package plus unused dependencies
pip-autoremove *package*remove a package plus unused dependencies (install pip install pip-autoremove first)

pip config​

commanddescription
pip config listshow all current settings. (There are multiple config files, see doc)

pip cache​

doc

commanddescription
pip cache -hhelp
pip cache dirshow cache folder

pipx​

  • pipx relies on pip (and venv)

pipx vs npx​

  • Both can run cli tools (npx will search for them in node_modules, and if not found run in a temporary environment. pipx run will search in __pypackages__ and if not found run in a temporary environment)
  • npx works with JavaScript and pipx works with Python
  • Both tools attempt to make running executables written in a dynamic language (JS/Python) as easy as possible
  • pipx can also install tools globally; npx cannot
    • npx looks into the local /node_modules folder for the package and if it can't find it, it will download and run it without having that package globally installed. source

venv (python3)​

  • Warning: Do not move or copy the env folder to other locations! Always re-create environments by using python3 -m venv env and re-installing all packages from a requirements.txt file.
commanddescription
python3 -m venv envcreate the environment env
source env/bin/activateactivate the environment env
deactivatedeactivate the environment that is currently activated
rm -r env/delete the environment env

virtualenv (python2)​

commanddescription
virtualenv -p /home/username/opt/python-2.7.15/bin/python venvcreate a virtualenv with name "venv"
  • the rest is similar to venv

requirements.txt files​

commanddescription
pip install -r requirements.txtinstall all packages from a requirements.txt file

pip freeze​

  • pip freeze has some issues
    • a better tool for generating requirements.txt: pip-compile
commanddescription
pip freeze > requirements.txtwrite all packages in the current environment to a requirements.txt file (Note: freeze vs. pipreqs: freeze saves all packages in the environment including those that you don't use in your current project!)

Problems:

  • pkg_resources==0.0.0
    • pip freeze also includes pkg_resources==0.0.0 in the requirements.txt which can cause errors, when running pip install -r requirements.txt

pipenv​

  • Warning: not much development here, i.e. maybe official support ends soon? Use poetry instead !
commanddescription
pip install --user pipenvPython's officially recommended packaging tool
pipenv installinstall right python version, environment and all dependencies
pipenv lock --clearif ERROR: No matching distribution found for markupsafe==1.0 ERROR: Couldn't install package: MarkupSafe Package installation failed... (and after this command run pipenv install again)
pipenvshows help
pipenv shellactivates env (similar to source /env/bin/activate for pip)
pipenv --rmremove the virtualenv created under /home/bra-ket/.local/share/virtualenvs
pipenv run python blockchain.pySpawns a command installed into the virtualenv.
pipenv graphshows installed dependencies

poetry​

Install poetry​

commanddescription
`curl -sSL https://install.python-poetry.orgpython3 -`
pip3 install poetry==1.1.15install poetry

Basics​

Activate the virtual environment (like source env/bin/activate):

cd path/to/poetry/project/folder/   # the folder that contains the files "pyproject.toml" and "poetry.lock"
poetry env use 3.11   # activate the right python version (only works if you have the python executable in your PATH)
poetry install   # install all packages in a poetry virtual environment
poetry shell   # activate the virtual environment
...            # now you can use all packages that are installed in the poetry virtual environment
exit           # deactivate the virtual environment (shortcut: ctrl-d)

Alternatively, you can use all packages that are installed in the poetry virtual environment by using poetry run, python-poetry.org, like so:

cd path/to/poetry/project/folder/   # the folder that contains the files "pyproject.toml" and "poetry.lock"
poetry env use 3.11   # activate the right python version (only works if you have the python executable in your PATH)
poetry install
poetry run python main.py -d /media/bra-ket/INTENSO/path/to/destination/folder/ https://domain.xyz/from/which/to/download/
commanddescription
poetry env infoshow information about currently active environment
poetry env listshow available environments
poetry env use someenvswitch environment (note: use pyenv shell some_python_version to switch python versions)
poetry env remove test-O3eWbxRl-py3.7delete existing virtual environments
commanddescription
poetry installreads the pyproject.toml file from the current project, resolves the dependencies, and installs them.
poetry install --no-devlike poetry install, but do not install the development dependencies
poetry showlike pip list

conda​

Vorinstallierte modules: List

commanddescription
conda config --set auto_activate_base Falseconda base environment nicht mit shell starten
conda env list
conda info --envs
conda create --name myenv
conda env create -f environment.yml
conda activate myenv
conda install --file requirements.txt
conda install -c conda-forge xlwings --override-channels--override-channels: ensures that conda searches only your specified channel (here: conda-forge) and no other channels, such as default channels or any other channels you may have listed in your .condarc file.
conda deactivate
conda remove --name myenv --all

CLI​

man python:

-c command
       Specify the command to execute (see next section).  This  termi‐
       nates the option list (following options are passed as arguments
       to the command).
-m module-name
       Searches  sys.path for the named module and runs the correspond‐
       ing .py file as a script. This terminates the option list  (fol‐
       lowing options are passed as arguments to the module).

Examples:

  • python3 -c "print('hello')"

Syntax​

relative imports​

  • if there is a from .somemodule import somefunc (the dot is important!) in a Python file somepackage/somefile.py, you have to run it via python3 -m somepackage.somefile or else the relative import will fail

ternary​

  • ternaries must have an else statement!

Check if variable exists​

if 'myVar' in locals():
  # myVar exists.

if 'myVar' in globals():
  # myVar exists.

if hasattr(obj, 'attr_name'):
  # obj.attr_name exists.

Exception​

All built-in Exceptions: Exception hierarchy

Do not use except Exception as e:​

"why not use except Exception as e"

Great question — this is one of those subtle Python practices that separates robust code from fragile code.

Let’s unpack why you shouldn’t (usually) use except Exception as e: by default, and when it is appropriate.

⚠️ Why you should be careful with except Exception as e​
1. It can hide bugs you didn’t expect​

except Exception catches all exceptions that inherit from Exception — and that includes many you might not want to handle, such as:

  • TypeError (often a programming mistake)
  • ValueError
  • AttributeError
  • KeyError
  • IndexError

So if you do:

try:
    do_something()
except Exception as e:
    print("Error:", e)

You might silently swallow real bugs like misspelled variable names, logic errors, or broken API usage.

✅ Better: Catch only what you expect.

try:
    data = json.loads(payload)
except json.JSONDecodeError as e:
    raise ValueError("Invalid JSON payload") from e
2. It can make debugging much harder​

When you catch Exception, you might prevent useful tracebacks from surfacing.

✅ Good:

try:
    risky_operation()
except ConnectionError as e:
    logger.error("Network issue: %s", e)
    raise  # re-raise for visibility

❌ Bad:

try:
    risky_operation()
except Exception as e:
    logger.warning("Something failed: %s", e)
    # silently continues — now your app behaves strangely
3. It does not catch critical exceptions properly​

except Exception does not catch:

  • SystemExit
  • KeyboardInterrupt
  • GeneratorExit

That’s good — those are meant to stop the program. But using a bare except: (no type) will catch those, which is even worse!

So:

except Exception:
    # safe(ish), doesn’t catch exit/interrupt

vs.

except:
    # 🚫 very bad: catches KeyboardInterrupt, SystemExit, etc.
4. It’s sometimes acceptable — but only at top-level boundaries​

If you’re writing:

  • A command-line app
  • A web server request handler
  • A background job loop

Then using except Exception as e: at the outermost layer makes sense — so you can log or report any unexpected error.

✅ Example:

def main():
    try:
        run_server()
    except Exception as e:
        logger.exception("Unhandled exception in main loop")
        sys.exit(1)

Here, you want to catch everything, log it, and exit cleanly.

🧭 General Rules of Thumb​
SituationBest Practice
You know the possible error typesCatch those explicitly
You want to convert one type to anotherUse raise ... from ...
You’re in a top-level handler (e.g., CLI, API endpoint)except Exception as e: is OK for logging
Inside library or utility codeAvoid except Exception — too broad
Need to debugNever swallow exceptions silently
✅ Example: Good structure​
def process_file(path):
    try:
        with open(path) as f:
            return json.load(f)
    except FileNotFoundError as e:
        raise RuntimeError(f"Missing file: {path}") from e
    except json.JSONDecodeError as e:
        raise ValueError(f"Invalid JSON in {path}") from e

def main():
    try:
        result = process_file("config.json")
        print(result)
    except Exception as e:
        logger.exception("Unhandled exception in main")
        sys.exit(1)
TL;DR​

Don’t use except Exception as e: unless:

  • You’re at the top level (logging, cleanup, graceful exit), or
  • You have a good reason to catch all operational errors.

Otherwise, catch specific exceptions so you don’t accidentally hide bugs.

Would you like me to show an example of how to safely log exceptions globally (using sys.excepthook or a decorator) instead of sprinkling except Exception everywhere?

raise​

"python frequent use of raise best practice"​

Good question — using raise properly in Python is key to writing clean, maintainable, and debuggable code. Below are best practices, patterns, and examples for frequent and effective use of raise.

🔹 1. Use specific exception types​

Always raise the most specific built-in or custom exception possible, not just Exception.

✅ Good:

if not isinstance(age, int):
    raise TypeError("age must be an integer")

❌ Bad:

raise Exception("Invalid input")  # too generic

This makes it easier for callers to catch and handle only the exceptions they expect.

🔹 2. Use raise without arguments to re-raise​

When you catch an exception and need to rethrow it after partial handling, use plain raise.

✅ Good:

try:
    process_data()
except ValueError as e:
    logger.error(f"Data processing failed: {e}")
    raise  # preserves original traceback

❌ Bad:

except ValueError as e:
    raise e  # loses original traceback context
🔹 3. Include informative messages​

Exception messages should tell what went wrong and why (include variable names or values when safe).

✅ Good:

if balance < 0:
    raise ValueError(f"Balance cannot be negative: {balance}")

❌ Bad:

raise ValueError("Invalid balance")
🔹 4. Define custom exceptions for domain-specific errors​

When your code represents a specific domain (e.g., finance, networking, etc.), create exceptions that express meaning clearly.

✅ Good:

class InsufficientFundsError(Exception):
    """Raised when an account withdrawal exceeds available balance."""
    pass

def withdraw(account, amount):
    if amount > account.balance:
        raise InsufficientFundsError(f"Cannot withdraw {amount}, balance is {account.balance}")

This allows targeted exception handling:

try:
    withdraw(user_account, 500)
except InsufficientFundsError:
    notify_user("Insufficient funds.")
🔹 5. Don’t use raise for flow control​

Avoid using exceptions for logic that isn’t exceptional.

❌ Bad:

try:
    value = next(iterator)
except StopIteration:
    value = None  # just use default behavior instead

✅ Good:

value = next(iterator, None)
🔹 6. Chain exceptions when relevant (raise ... from ...)​

Use raise ... from ... to preserve context when you raise a new error in response to another one.

✅ Good:

try:
    config = load_config(file_path)
except FileNotFoundError as e:
    raise RuntimeError("Failed to initialize system: missing config") from e

This helps debugging by showing both the original and new exception traces.

🔹 7. Use assertions only for internal sanity checks, not user errors​

Use raise for user-facing validation; use assert for developer invariants.

✅ Good:

if not user_input:
    raise ValueError("Input cannot be empty")

✅ Also Good (for internal checks):

assert total >= 0, "Internal logic error: total cannot be negative"
🔹 8. Document raised exceptions​

When writing functions or APIs, document what exceptions might be raised.

✅ Example (docstring):

def divide(a, b):
    """
    Divide two numbers.
    
    Raises:
        ZeroDivisionError: If b is zero.
        TypeError: If a or b are not numbers.
    """
    if not all(isinstance(x, (int, float)) for x in (a, b)):
        raise TypeError("Both arguments must be numeric")
    return a / b
🔹 9. Validate early, fail fast​

Raise exceptions as soon as possible when invalid input is detected — makes bugs easier to locate.

✅ Good:

def create_user(name, age):
    if not name:
        raise ValueError("Name is required")
    if age < 0:
        raise ValueError("Age cannot be negative")
🔹 10. Log or handle exceptions at boundaries​

Inside core logic: raise. At program or API boundaries: handle and log.

✅ Pattern:

def main():
    try:
        run_server()
    except Exception as e:
        logger.exception("Unhandled error in main loop")
        sys.exit(1)
🧭 Summary​
Best PracticeWhy It Matters
Use specific exception typesImproves handling and clarity
Preserve traceback with bare raiseKeeps debugging context
Write informative messagesAids debugging
Create custom exceptionsExpress domain intent
Don’t use exceptions for control flowAvoids performance & logic confusion
Use raise ... from ...Maintains causal chain
Document raised exceptionsImproves API usability
Fail fastEasier debugging

Would you like me to show a short reusable template (like a Python “error handling boilerplate”) that applies these best practices?

Bare raise inside except clause​

stackoverflow

When inside an except clause, you might want to, for example, log that a specific type of error happened, and then re-raise. The best way to do this while preserving the stack trace is to use a bare raise statement. For example:

logger = logging.getLogger(__name__)

try:
    do_something_in_app_that_breaks_easily()
except AppError as error:
    logger.exception(error)
    raise                 # just this!
    # raise AppError      # Don't do this, you'll lose the stack trace!

logger.error() vs logger.exception()​

MethodLogs whatIncludes traceback?When to use
logger.error()Your custom error message❌ No traceback by defaultGeneral errors, not necessarily from exceptions
logger.exception()Error message + full traceback✅ Yes, automatically (if inside an except block)Inside an except block when an exception was raised

logger.exception() is essentially a shortcut for:

logger.error("message", exc_info=True)

So it automatically includes the exception traceback that Python keeps in sys.exc_info().

It only works properly inside an except block — if you call it outside one, it won’t have any exception info.

Don't raise generic Exceptions​

raise Exception('I know Python!') # Don't! If you catch, likely to hide bugs.

Instead, use the most specific Exception constructor that semantically fits your issue. → Exception hierarchy

Patterns​

if not type(x) is int:
  raise TypeError("Only integers are allowed")

repl​

Start flags​

commanddescription
python3 -vzeige Details der ausgeführten Befehle (eg. automatische imports, andere getriggerte Befehle, Konstruktor calls, etc.)

package source code location​

commanddescription
pip show torchshow the location of package "torch" (there you can find the source code of the package)

Useful Packages​

argparse​

argparse

  • erlaubt zB mit python prog.py 1 2 3 4 5 —sum einen command line Befehl selber zu definieren:
    • The argparse module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and argparse will figure out how to parse those out of sys.argv. The argparse module also automatically generates help and usage messages and issues errors when users give the program invalid arguments.

functools​

partial()​

new_function = partial(some_function, *args)

  • functools.partial documentation
  • definiert eine neue Funktion new_function, die genau das gleiche macht wie some_function
    • Praktisch, um bestimmte Argumente einer Funktion festzulegen, damit sie nicht wieder eingegeben werden müssen: zB basetwo = partial(int, base=2) um nicht jedes mal base=2 eingeben zu müssen um binäre Zahlen in Dezimalzahlen umzuwandeln

sys​

  • sys.path
    • doc
    • A list of strings that specifies the search path for modules. Initialized from the environment variable PYTHONPATH, plus an installation-dependent default.

setuptools​

setup.py​

protobuf​

  • WARNING: each protobuf version runs only under specific Python versions, install it via python3.Y -m pip install protobuf=X.Y.Z
    • otherwise you will get the error: protobuf requires Python '>=3.7' but the running Python is 3.6.4, stackoverflow

pathlib​

import: from pathlib import Path

pathlib vs os.path

"pathlib is the more modern way since Python 3.4."

descriptionpathlib methodos.path method
get the "stem" (filename without extension)path.stemsplitext(basename(path))[0]
get just the filenamep.name
get absolute pathsp.resolve()

Coding Tricks​

Python Script cannot be killed​

ps ax | grep python, find the PID of your running script and then run kill <PID>

or: ps ax | grep python | cut -c1-5 | xargs kill -9

Using C++​

Create Python bindings of existing C++ code​