Python
https://rednafi.github.io/digressions/python/2020/07/03/python-mixins.html
https://realpython.com/courses/python-development-visual-studio-code-setup-guide/
https://devblogs.microsoft.com/python/
https://gto76.github.io/python-cheatsheet/
https://thegurus.tech/posts/2019/05/hadoop-python/
https://habr.com/ru/post/454604/ python pointers
https://habr.com/ru/company/otus/blog/448350/ . How Python dictionary implemented
https://treyhunner.com/2019/05/python-builtins-worth-learning/
https://datawhatnow.com/things-you-are-probably-not-using-in-python-3-but-should/
https://www.youtube.com/channel/UCxs2IIVXaEHHA4BtTiWZ2mQ/videos . PyCon 2019
How to start python http server:
python -m SimpleHTTPServer <8000> (python2)
python3 -m http.server <8000> (python3)
http://192.168.1.2:8000
http://127.0.0.1:8000
https://github.com/gto76/python-cheatsheet
http://grishaev.me/interview/
https://julien.danjou.info/
CPython compiles the python code into the byte code (myfile.py → mifile.pyc).
Then the python interpreter consumes the byte code and act on it.
For performance reasons Python doesn't recompile every time,
so it caches the content of the compiled code.
https://indianpythonista.wordpress.com/2018/01/05/demystifying-pyc-files/
Python can be invoked from the command line with several flags;
you can find the flags via
python --help
https://dbader.org/blog/records-structs-and-data-transfer-objects-in-python
https://github.com/vinta/awesome-python
All pairs similarity search
https://github.com/ekzhu/SetSimilaritySearch/blob/master/scripts/all_pairs.py
https://news.ycombinator.com/item?id=18483859
slots
https://habr.com/post/427909/ . __slots__
https://blog.usejournal.com/a-quick-dive-into-pythons-slots-72cdc2d334e
http://book.pythontips.com/en/latest/__slots__magic.html
Modules and Packages
Big application usually contains many python files, the import statement allows to glue the code.
import mod
When the interpreter executes the above import statement, it searches for mod.py in a list of directories assembled from the following sources:
The directory from which the input script was run or the current directory if the interpreter is being run interactively
The list of directories contained in the PYTHONPATH environment variable, if it is set (not recommended)
An installation-dependent list of directories configured at the time Python is installed
If a file named __init__.py is present in a package directory, it is invoked when the package or a module in the package is imported.
The resulting search path is accessible in the Python variable sys.path which you can print:
python -c "import os, sys; print(os.linesep.join(sys.path))"
https://realpython.com/python-modules-packages/
Package managers: pip and conda
pip check . # test consistency of current setup
https://snarky.ca/why-you-should-use-python-m-pip/
https://news.ycombinator.com/item?id=21429250
pipx
https://jacobian.org/2019/nov/11/python-environment-2020
https://towardsdatascience.com/how-to-setup-an-awesome-python-environment-for-data-science-or-anything-else-35d358cc95d5
PIP package manager: https://pypi.org/project/pip/
https://dev.to/elabftw/stop-using-sudo-pip-install-52mn
The 3rd-party libraries installed by pip are usually here:
python -c "import os, site; print(os.linesep.join(site.getsitepackages()))"
Why you should use python -m pip , not just pip
https://habr.com/ru/company/otus/blog/475392/
python3.8 -m pip
pip commands:
pip -V # shows pip version and path (make sure it in sync with output of site.getsitepackages()) above)
pip list # all installed packages
pip show # the package details
pip install
conda is similar to pip - it goes with Anaconda Python distribution https://conda.io/docs/user-guide/overview.html
Python 2 vs Python 3
Integer division:
7/5 # returns 1 for Python 2; returns 1.4 for Python3
Print statement:
print 'A' # works for Python2 only
print ('A') # works for Python2 and Python3
In Python 2, implicit str type is ASCII. But in Python 3.x implicit str type is Unicode.
xrange() exists in Python2 only; under Python3 it is named range()
jteppinette 44 minutes ago [-]
No compatibility layer. Its really not bad.
there are a few modules that are simply at different locations but have the same API
PY3 = sys.version_info >= (3, 0)
PY2 = sys.version_info < (3, 0)
PY26 = sys.version_info >= (2, 6) and sys.version_info < (2, 7)
if PY3:
from http import client as httplib
else:
import httplib
is string:
isinstance(, basestring if PY2 else str)
using different classes
# Python 2.6 doesn't properly UTF-8 encode syslog messages, so it needs
# to be performed in a custom formatter.
formatter_class = UnicodeLoggingFormatter if PY26 else logging.Formatter
Enforcing named parameters (Python3):
-----------------------------------
def fn(*, a = 100,b = 200):
return a+b*10
Variadic Functions
------------------
def fn(a ,*all ):
sum=0;
for item in all:
sum+=item
return a+sum
print(fn(10,20)) # 30
Keyword Parameters
--------------------
def fn(**kwargs ):
return kwargs['a'] + kwargs['b']
print(fn(a=200,b=500))
d1 = {'a':100 , 'b':200}
print(fn(**d1)) # 300
To change global variables
---------------------------
num = 9
def f1():
global num
num = 20
Closure
--------
one can pass a function as a parameter to another function or return a function from another function.
def getmulby(m):
def op(n):
return m*n
return op
f1=getmulby(10)
f2=getmulby(5)
print( f1(2) ) # 20
print( f2(2) ) # 10
Decorator
----------
Decorator is a good example of closure. Decorator is a function that takes another function and extends
the behavior of the latter function without explicitly modifying it
def add_stars(some_function):
def wrapper():
print("********************")
some_function()
print("********************")
return wrapper
@add_stars
def my_function():
print("Hello!!!")
my_function()
# ********************
# Hello!!!
# ********************
## list.sort() vs sorted()
nums = [1, -3, 2, 5]
new_nums = sorted(nums)
print(new_nums) # -3, 1, 2, 5
print(sorted(nums, reverse=True)) # 5, 2, 1, -3
# abs is the built in absolute value function
print(sorted(nums, key=abs)) # 1, 2, -3, 5
nums = [1, 2, 3]
for i in reversed(nums): # <- reversed
print(i) # 3, 2, 1
# sum() min() max()
def average(nums):
return sum(nums)/len(nums)
def range(nums):
return max(nums)-min(nums)
# Enumerate()
for index, num in enumerate(nums):
# Sort the words in the string based on length:
sorted(s.split(), key=len)
# Join list elements into string:
words = ["Hello", "World"]
combined = " ".join(words)
Lambda Expressions
-------------------
ls = [2,4,6]
newlist = map(lambda item:item * 2, ls)
for n in newlist:
print(n)
Virtual Environments
https://github.com/pipxproject/pipx
mksir ~/.venvs
For Python2
To create: virtualenv ~/.venvs/myproject
To activate: ~/.venvs/myproject/bin/activate
For Python3
To create: python3 -m venv ~/.venvs/myproject
To activate: . ~/.venvs/myproject/bin/activate
Python virtual env
Python virtual environment
When you create a virtual environment, all you are doing is creating a copy of the Python executable,
some core modules in the dist-packages folder,
and (usually) an empty site-packages folder. That it.
python -c "import sys; print(sys.executable)"
Using virtual environments allows you to avoid installing Python packages globally which could break system tools or other projects.
There are several tools to manage the virtual environments: https://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe/41573588#41573588
The virtualenv is the basic tool which works with Python2 and Python3: https://virtualenv.pypa.io/en/stable/
By default, Virtualenv creates virtual environment using the version of Python under which it is installed. To specify another version of the Python use the -p option.
On some occasions, you might want to create a virtual environment with the packages from the global Python installation. This can be accomplished using --system-site-packages option.
virtualenv v1 # create new virtual environment named v1; it will have the local folders: bin, include and lib
source v1/bin/activate # activate it; it changes the $PATH so that the bin/directory of the virtual environment will become first in the list
echo $PATH # it is different now
which python
pip install ... # packages will be installed into v1/lib/pythonX.Y/site-packages
deactivate # deactivate
-- pyenv
https://github.com/pyenv/pyenv
https://axcoto.com/notes/manage-python-versions-on-macosx/
https://www.marc-richter.info/using-pyenv-to-manage-your-python-interpreters/
https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html
conda-env list
# conda environments:
#
base * /Users/miclub01/anaconda3
tensorflow /Users/miclub01/anaconda3/envs/tensorflow
tensorflow_env /Users/miclub01/anaconda3/envs/tensorflow_env
conda activate tensorflow
http://pycon.ru/
My Python code snippets
https://python-patterns.guide/
https://habr.com/company/it_people/blog/422363/ Videos from Russian Python Conf
https://github.com/faif/python-patterns
Memory management
https://habr.com/ru/company/otus/blog/443312/ .
https://habr.com/ru/post/455722/
https://pythonprogramming.net
https://pythonawesome.com/
https://pythondigest.ru/
https://pynative.com/python-postgresql-tutorial/ Postgres
https://rushter.com/blog/numba-cython-python-optimization/
https://hackernoon.com/python-tricks-101-2836251922e0
https://habr.com/post/421993/
Asyncio
https://realpython.com/async-io-python/
https://habr.com/ru/post/453348/
https://youtu.be/pIXiChn5j4E
https://www.techiegeek.eu/?p=25
http://theautomatic.net/2017/09/29/downloading-every-file-ftp-server/
https://www.youtube.com/watch?time_continue=187&v=WiQqqB9MlkA . Nina Zakharenko - Elegant Solutions For Everyday Python Problems - PyCon 2018
https://stackoverflow.com/questions/45684618/having-default-mac-python-2-7-and-anaconda-python-3
https://tech.instacart.com/freezing-pythons-dependency-hell-in-2018-f1076d625241
alex.dzyoba.com/blog/python-import/ import
https://dwhsys.com/2017/01/23/working-with-multiple-python-versions/
pip check
python -c "import os, sys; print(os.linesep.join(sys.path))"
How to see where module is located in file system:
import six
print six.__file__
PYTHONPATH
In order to let python search first the most updated version of certain package,
instead of removing the system version, what can be done is to set the system variable
PYTHONPATH in the ~/.bash_profile (or ~/.bashrc if linux) config file to the path where the new packages are installed:
export PYTHONPATH=/Library/Python/2.7/site-packages
An alternative is to modify the python path inside your python script by adding the path at the beginning of the path list:
import sys
sys.path.insert(1,'/Library/Python/2.7/site-packages')
This needs to be done for every script you need a certain package version.
You might want for some reason use an older version that you have installed.
Random notes
assert hasattr(Base,'foo'), "no method foo in Base class"
Можете запустить ее предварительно — strace python script.py — но обычно удобнее подключаться к уже работающему приложению: strace -p PID.
$ cat test.py
with open('/tmp/test', 'w') as f:
f.write('test')
$ strace python test.py 2>&1 | grep open | tail -n 1
open("/tmp/test", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3
Algorithms in Python
Python cheatsheet
https://habrahabr.ru/post/349860/ regexp
http://overapi.com/python
https://opensource.com/article/17/4/grok-gil
https://www.youtube.com/watch?v=7lmCu8wz8ro
https://www.youtube.com/watch?v=MCs5OvhV9S4 Python concurrency
https://www.amazon.com/Python-Data-Analysis-Wrangling-IPython/dp/1491957662 2nd edition
https://www.amazon.com/Python-Tricks-Buffet-Awesome-Features/dp/1775093301
https://www.amazon.com/Dynamic-Programming-Coding-Interviews-Bottom-Up/dp/1946556696
https://www.hardikp.com/2017/12/30/python-cpp/
HTTP Server
How to start http server:
python2 -m SimpleHTTPServer
python3 -m http.server 8000
http://192.168.1.2:8000
http://127.0.0.1:8000
If the directory has a file named index.html, that file will be displayed.
If there is no index.html, then the files in the directory will be listed.
If you wish to change the port from default (8000) then specify it explicitly:
python -m SimpleHTTPServer 8080
Random Notes
http://arseny.info/2017/parallel-computation-with-two-lines-of-code.html Joblib
https://www.dataquest.io/blog/python-counter-class/
https://github.com/ChrisKnott/Eel offline web based UI for python
a='12345'
a[::-1] # reverse string
a[-1] # 5 - last item in the array equivalent to slice(-1)
a[-2:] # 45 - last two items in the array equivalent to slice(-2)
a[:-2] # 123 - everything except the last two items equivalent to slice(None, -2)
# List traversal
range(start, stop, hop)
range(n) # [0,1,...,n-1]
range(1,n) # [1,...,n-1]
range(1,n,2) # [1,3,5,...,n-1] if n is even, or [1,3,5,...,n-2] if n is odd
range(n,-1,-1) # [n,n-1,n-2,...,0]
range(len(arr)) # Provides indices of an array arr
range(len(arr)-1,-1,-1) # Provides indices of arr backwards
# List slicing
arr[w:s] # Wait w elements, start copy (:), stop before reaching index s
arr = [1,2,3,4]
arr[1:] = [2,3,4]
arr[:2] = [1,2]
# List manipulation
arr = [1,2,3]
[str(x) for x in arr] # Output: ['1','2','3']
map(lambda x: str(x), arr) # Output: ['1','2','3']
[str(x) for x in arr if x%2] # Output: ['1','3']
# List as queue
arr = [1,2,3]
arr.append(x) # queue.push(x)
arr.pop(0) #queue.pop() . removes the 1st element from array
arr[0] #queue.peek()
# List as stack
arr = [1,2,3]
arr.append(x) #stack.push(x)
y = arr.pop() # stack.pop() . removes the last element from list
arr[-1] # stack.peek()
Dictionary
for k, v in dict.items():
print(k,v)
d.keys() d.values() . d.items() . this is Dictionary Views in Python 3
d.iterkeys() . d.itervalues() d.iteritems()
Default dictionary
from collections import defaultdict
d = defaultdict(list)
d['python'].append("awesome")
d['something-else'].append("not relevant")
d['python'].append("language")
for i in d.items():
print i
Match brackets
expect = {"(":")","[":"]","{":"}"}
def wellformed(string):
stack = []
for char in string:
if char in expect.keys():
stack.append(char)
elif char in expect.values():
if not stack or expect[stack.pop()] != char:
return False
return not stack
print(wellformed("([])[]({})")) # True
print(wellformed("([)]")) # False
print(wellformed("((()")) # False
multiply a vector by a scalar:
def scale(A, x): return [ai*x for ai in A]
add 2 vectors:
def add(A, B): return [ai+bi for (ai, bi) in zip(A, B)]
dot product:
def dot(A, B): return sum([ai*bi for (ai, bi) in zip(A, B)])
matrices by vector multiplication:
def mul(A, X): return [dot(ai, X) for ai in A]
Counter usage
from collections import Counter
def is_anagram(word1, word2):
return Counter(word1) == Counter(word2)
print(is_anagram('tachymetric', 'mccarthyite'))
print(is_anagram('banana', 'peach'))
------------- print most frequent words in file --------------
import re
from collections import Counter
def extract_words(filename):
text = open(filename).read().lower()
return re.findall('\w+', text)
def filter_words(words, ignore=None):
if not ignore:
ignore = set(['is','if','a','it','the','an','in','of','to','and','that','be',
'his','he','her','on','not','by','s','ch','are','this','as','for',
'was','with','which','or','for','from','i','you','at','when','have',
'but','may','they','their','be','who','your','says','said','all',
'him','1','2','3','4','5','6','7','8','9','t','o','-','_'])
return [word for word in words if word not in ignore]
def count_words(words, limit=10, ignore=None):
filtered = filter_words(words, ignore)
top_words = Counter(filtered).most_common(limit)
return top_words
def show_top_words(filename, limit=10):
words = extract_words(filename)
for word, frequency in count_words(words, limit):
print("%15s : %s" % (word,frequency))
if __name__ == "__main__":
filename = sys.argv[1]
limit = int(sys.argv[2])
show_top_words(filename, limit)
--- polinom ----------
class Polynom:
def __init__(self, *coefs):
self.coefs=coefs
def __call__(self):
....
def __repr__(self):
return "Polynom (*{!r})".format(self.coefs)
def __add__(self, other):
return Polynom(* (x+y) for x,y in zip(self.coefs, other.coefs))
-- dictionary -----
a={'k1':'v1', 'k2':'v2'}
for key, val in a.iteritems():
print key, val
fruits = { 'apple':1, 'orange':2, 'banana':3 }
if 'apple' in fruits:
print(fruits['apple'])
--- Exceptions ---
def linux_interaction():
assert ('linux' in sys.platform), "Function can only run on Linux systems."
print('Doing something.')
try:
linux_interaction()
except AssertionError as error:
print(error)
else:
try:
with open('file.log') as file:
read_data = file.read()
except FileNotFoundError as fnf_error:
print(fnf_error)
finally:
print('Cleaning up, irrespective of any exceptions.')
-- Fibonacci ---------
def fib(num):
a,b=0,1
for i in xrange(0,num):
yield "{}: {}".format(i+1,a)
a,b = b, a+b
for item in fib(10):
print item