Python General Development Tools Index

Environment Manager

Environment Manager

1. conda
2. venv
   1. used by:
      1. uv
      2. rye
3. mamba
4. poetry

Link to original

Making Packages

Making Packages

1. Pipenv,
2. pip-tools,
3. PDM
4. Poetry
5. rye
6. uv

Pipenv, pip-tools, PDM, or Poetry? : r/Python (reddit.com)

https://github.com/astral-sh/rye/discussions/6#discussioncomment-7454958

rye uses pip-tools and create platform dependent lock file

poetry create platform independent lock file

https://star-history.com/#indygreg/python-build-standalone&pdm-project/pdm&astral-sh/rye&python-poetry/poetry&pypa/hatch&pypa/pipenv&astral-sh/uv&jazzband/pip-tools&pypa/pip&Date

uv is growing to become a full replacement of rye uv is used by rye uv is catching up to poetry in terms of github stars

Link to original

Testing Packages

Testing Packages

python testing for multiple python environment versions

• tox

python testing

• pytest

Link to original

Publishing Packages Methods

Publish Location

• PyPi
• anaconda

Publishing procedure

• prior to publishing the project should undergo code linting and compilation. Of course all this can be done locally. but it’s best to automate it using github workflow.
• some GitHub workflow actions that’s worth looking include:
  • https://github.com/astral-sh/setup-uv
  • https://github.com/eifinger/setup-rye?tab=readme-ov-file
  • https://github.com/marketplace/actions/python-setup-rye
  • https://github.com/marketplace/actions/pypi-publish The Official Github Actions has a workflow for Publishing Python Package. It uses the action pypi-publish. Inside which contains a reference to the following document
• https://docs.github.com/en/actions/use-cases-and-examples/building-and-testing/building-and-testing-python#publishing-to-package-registries

Link to original

Diagrams as Code

Diagrams as Code

potential diagrams inmarkdown

Commonmark out of box doesn't seem to support any diagram render capability. Hence everything here is targeting some flavour of markdown.

• prismjs (not a diagram but do syntax highlighting of the code itself)
• plantuml (really old and mature)
• mermaid (avail in github and gitlab)
• d2
  • (need local install compiler to generate visual, not supported out of the box)
  • have a lot of additional features than mermaid
  • https://text-to-diagram.com/?example=code&b=mermaid
  • https://github.com/terrastruct/d2?tab=readme-ov-file#install
• https://kroki.io/
  • this seems to be a website (with self host option) that offer an api call to convert encoded diagram strings to visual image for a lot of different visualization DSL
• Terraform outputs plots in graphviz DOT format.
  • below 2 source describe a way to use external/self hosted services to embed the diagram in markdown as an api call to website.
    • http://bijanebrahimi.github.io/blog/graphviz-in-markdown.html
    • https://github.com/TLmaK0/gravizo
    • the two source describes calling web api with multiline url. Not sure if that ever worked but currently it doesnt in github markdown nor quartz compilation. Also gravizo’s readme shows a different method of feeding the link of the github readme into their website to generate the plot instead, but this is not viable cos the url for a digital garden SHOULD NOT be hardcoded.
  • looking to see if there are method to export diagram from Terraform in mermaid instead.
    • conversion tool from dot (graphviz) to mermaid not found.
    • https://github.com/RoseSecurity/Terramaid
      • custom install to convert terraform to mermaid (i.e. use this instead of terraform graph)
      • https://github.com/RoseSecurity/Terramaid/issues/25
        • does not support opentofu yet
    • https://github.com/hashicorp/terraform/issues/30519#issuecomment-2442465590
      • mentions
        • terramaid
        • sed approach
          • https://www.heaton.dev/2022/05/use-terraform-graph-in-mermaidjs/
  • https://kroki.io/ has api taking diagram encoded using deflate + base64 algorithm. This can act as a substitute until one learn how to use sed to do dirty conversion from dot to mermaid as described in
  • https://www.gravizo.com/ is another website with api that uses graphviz to render DOT, PlantUML and UMLGraph syntax diagrams
    • example
    • 
  • Obsidian has plugins that support graphviz. However, to render output from custom obsidian plugin feels complicated.
    • https://joschua.io/posts/2023/09/01/obsidian-publish-dataview
      • this is trying to:
        • use some api call of the dataview plugin to convert things into markdown,
        • embedded in any markdown doc,
        • and then use obsidian publish.

example of failing to rendering graphviz diagram in github readme due to multi line inside link


example of rendering the diagram via kroki.io

bash script that generate kroki link from .dot file(can be self hosted)

 
# syntax used here is bash parameter expansion
# https://stackoverflow.com/questions/2013547/assigning-default-values-to-shell-variables-with-a-single-command-in-bash
file="${1:-content/Learn/DevOps/example2.dot}"
cat $file | python -c "import sys; import base64; import zlib; print(base64.urlsafe_b64encode(zlib.compress(sys.stdin.read().encode('utf-8'), 9)).decode('ascii'))" | sed 's/^/ https:\/\/kroki.io\/graphviz\/svg\//'
 

example .dot file of terraform graph

digraph {
compound = "true"
newrank = "true"
subgraph "root" {
  "[root] aws_instance.example"
  [label = "aws_instance.example", shape = "box"]
  "[root] aws_security_group.instance"
  [label = "aws_security_group.instance", shape =a "box"]
  "[root] provider.aws"
  [label = "provider.aws", shape = "diamond"]
  "[root] aws_instance.example" ->
  "[root] aws_security_group.instance"
  "[root] aws_security_group.instance" ->
  "[root] provider.aws"
  "[root] meta.count-boundary (EachMode fixup)" ->
  "[root] aws_instance.example"
  "[root] provider.aws (close)" ->
  "[root] aws_instance.example"
  "[root] root" ->
  "[root] meta.count-boundary (EachMode fixup)"
  "[root] root" ->
  "[root] provider.aws (close)"
  }
}

result

Link to original

Documenting package

Documenting package

generate doc website for package from docstrings

• https://medium.com/practical-coding/documenting-your-python-library-from-zero-to-website-488f87ae58f5
• uses sphinx, gh-pages, github workflow

other options

• https://medium.com/@peterkong/comparison-of-python-documentation-generators-660203ca3804
• pdoc
• pydoctor (only python 2)
• doxygen
• pydoc (standard library)

https://www.reddit.com/r/Python/comments/lohyb/rpython_what_documentation_system_do_you_use_and/

readme standards

https://codingnomads.com/python-101-documentation-readme

Visualizing code

• code2flow

Link to original