PythonProgramming

1 PYTHON PROGRAMMING Notes by Michael Brothers, available on http://github.com/mikebrothers/data-science/ Content taken from the following sources (among others): Avinash Jain's Udemy course Introduction To Python Programming https://www.udemy.com/pythonforbeginnersintro/ Jose Portilla's Udemy course Complete Python Bootcamp https://www.udemy.com/complete-python-bootcamp/ Bill Lubanovic's book Introducing Python: Modern Computing in Simple Packages (O'Reilly Media; 1st edition, December 4, 2014) Table of Contents PYTHON PROGAMMING ........................................................................................................................................................ 4 Variable............................................................................................................................................................................... 4 Multiple Declaration .......................................................................................................................................................... 4 Multiple Assignment.......................................................................................................................................................... 4 Data Types.......................................................................................................................................................................... 4 Operators............................................................................................................................................................................ 4 Relational operators: ......................................................................................................................................................... 4 Chained Comparison Operators: ....................................................................................................................................... 4 Strings: ................................................................................................................................................................................ 4 Lists: .................................................................................................................................................................................... 4 Tuples:................................................................................................................................................................................. 4 Dictionaries:........................................................................................................................................................................ 4 Sets:..................................................................................................................................................................................... 5 Comments:.......................................................................................................................................................................... 5 WORKING WITH STRINGS ...................................................................................................................................................... 5 Built-in String Functions:.................................................................................................................................................... 5 Built-in String Methods:..................................................................................................................................................... 5 Splitting Strings: ................................................................................................................................................................. 5 Joining Strings:.................................................................................................................................................................... 5 Turning Objects Into Strings............................................................................................................................................... 6 Escape characters:.............................................................................................................................................................. 6 Placeholders: ...................................................................................................................................................................... 6 FORMAT.................................................................................................................................................................................. 7 WORKING WITH LISTS:........................................................................................................................................................... 8 Built-in List Functions:........................................................................................................................................................ 8 Built-in List Methods:......................................................................................................................................................... 8 List Index Method............................................................................................................................................................... 9 Making a list of lists:........................................................................................................................................................... 9 LIST COMPREHENSIONS ......................................................................................................................................................... 9 WORKING WITH TUPLES: .....................................................................................................................................................102 WORKING WITH DICTIONARIES:..........................................................................................................................................10 Dictionary Comprehensions:............................................................................................................................................10 WORKING WITH SETS:..........................................................................................................................................................10 Set Operators: ..................................................................................................................................................................10 Built-in Set Methods: .......................................................................................................................................................11 RANGE...................................................................................................................................................................................11 CONDITIONAL STATEMENTS & LOOPS ................................................................................................................................12 If / Elif / Else statements:.................................................................................................................................................12 For Loops ..........................................................................................................................................................................12 While Loops......................................................................................................................................................................12 Nested For Loops..............................................................................................................................................................12 Loop Control Statements (Break, Continue & Pass)........................................................................................................13 Try and Except..................................................................................................................................................................13 INPUT (formerly raw_input) ................................................................................................................................................13 UNPACKING ..........................................................................................................................................................................14 Tuple Unpacking...............................................................................................................................................................14 Dictionary Unpacking .......................................................................................................................................................14 FUNCTIONS ...........................................................................................................................................................................15 Default Parameter Values................................................................................................................................................15 Positional Arguments *args and **kwargs .....................................................................................................................15 Inner Functions:................................................................................................................... Error! Bookmark not defined. Closures: .............................................................................................................................. Error! Bookmark not defined. PRE-DEFINED FUNCTIONS ....................................................................................................................................................16 LAMBDA EXPRESSIONS ........................................................................................................................................................16 MORE USEFUL FUNCTIONS ..................................................................................................................................................17 MAP ..................................................................................................................................................................................17 REDUCE .............................................................................................................................................................................17 FILTER................................................................................................................................................................................17 ZIP .....................................................................................................................................................................................17 ENUMERATE .....................................................................................................................................................................18 ALL & ANY.........................................................................................................................................................................18 COMPLEX ..........................................................................................................................................................................18 PYTHON THEORY & DEFINITIONS ........................................................................................................................................19 FUNCTIONS AS OBJECTS & ASSIGNING VARIABLES ............................................................................................................20 FUNCTIONS AS ARGUMENTS ...............................................................................................................................................20 DECORATORS:.......................................................................................................................................................................21 GENERATORS & ITERATORS.................................................................................................................................................22 NEXT & ITER built-in functions:........................................................................................................................................22 GENERATOR COMPREHENSIONS .....................................................................................................................................22 WORKING WITH FILES ..........................................................................................................................................................23 READING AND APPENDING FILES.........................................................................................................................................23 RENAMING & COPYING FILES ..............................................................................................................................................233 OBJECT ORIENTED PROGRAMMING – Classes, Attributes & Methods..............................................................................24 MODULES..............................................................................................................................................................................28 COLLECTIONS Module: .........................................................................................................................................................28 Counter.............................................................................................................................................................................28 defaultdict ........................................................................................................................................................................29 OrderedDict......................................................................................................................................................................29 namedtuple ......................................................................................................................................................................30 DATETIME Module ...............................................................................................................................................................30 TIMEIT Module .....................................................................................................................................................................30 PYTHON DEBUGGER – the pdb Module...............................................................................................................................31 REGULAR EXPRESSIONS – the re Module............................................................................................................................31 Searching for Patterns in Text...........................................................................................................................................31 Finding all matches ...........................................................................................................................................................32 Split with regular expressions...........................................................................................................................................32 Using metacharacters.......................................................................................................................................................32 STYLE AND READABILITY (PEP 8) .........................................................................................................................................34 GOING DEEPER: ....................................................................................................................................................................36 The '_' variable .................................................................................................................................................................36 To print on the same line:................................................................................................................................................36 Some more (& obscure) built-in string methods:............................................................................................................37 Some more (& obscure) built-in set methods:................................................................................................................37 Common Errors & Exceptions: .........................................................................................................................................38 For more practice: ............................................................................................................................................................384 PYTHON PROGAMMING Variable: reserved memory space. Can hold any value, assigned to a term. Case-sensitive. Can’t contain spaces. Variable names: 1. Names can not start with a number. 2. There can be no spaces in the name, use _ instead 3. Can’t use any of these symbols: ' " , < > / ? | \ ( ) ! @ # $ % ^ & * ~ - + 4. It’s considered best practice (PEP8) that the names are lowercase. 5. Don't use these reserved words: and assert break class continue def del elif else except exec finally for from global if import in is lambda not or pass print raise return try while Multiple Declaration: var1, var2, var3 = 'apples','oranges','pears' Multiple Assignment: var1 = var2 = var3 = 'apples' (spaces/no spaces doesn’t matter) Data Types: number (integer or float), string (text), list, tuple, dictionary, set, Boolean (True, False, None) Operators: + - * / addition, subtraction, multiplication, division % modulo = gives the remainder after division // floor divisor = discards the fraction without rounding ** exponentiator >>> 5/2 returns 2.5 NOTE: Python 2 treats '/' as 'classic division' >>> 5//2 returns 2 and truncates the decimal. Python 3 does >>> 5%2 returns 1 'true division' and always returns a float. >>> 5**3 returns 125 Note: 2 is an int type number, while 2.5 is a float. Division returns a float. (6/3 returns 2.0) Relational operators: (aka Comparison Operators) > greater than >= greater than or equal to < less than <= less than or equal to == equal to (use == when comparing objects. != not equal to One equals sign is used to assign values to objects.) <> not equal to Chained Comparison Operators: 1 < 2 < 3 returns True (this is shorthand for 1 < 2 and 2 < 3) Strings: anything between two sets of quotation marks (single or double) use \n in a string to insert a line-break, \t for a tab NOTE: strings are immutable. You can’t change elements in a string once they’re created, but you can add to them Lists: list1 = ['apples','oranges','pears'] created using square brackets Tuples: tuple1 = (1,2,3) created using parentheses Tuple elements cannot be modified once assigned (tuples are immutable) max(tuple1) returns 3, min(tuple1) returns 1 Strings, lists and tuples are sequences. Their contents are indexed (0,1,2…) list1[1] returns 'oranges' tuple1[1] returns 2 Dictionaries: contain a key and a value, using { } and colons dict1 = {'Tom':4, 'Dick':7, 'Harry':23} created using curly braces dict1['Harry'] returns 23 Dictionaries are mappings, not sequences. dict1[1] would return an error.5 Sets: behave like dictionaries, but only contain unique keys. Sets are unordered (not sequenced). set1 = set([1,1,2,2,3]) this is called "casting a list as a set" set1 returns {1,2,3} Comments: # (hash) provides quick one-liners """ (triple quotes) allow multiline full text (called docstrings) """ WORKING WITH STRINGS Slices: var1[10] returns the 11th character in the string (all indexing in Python starts at 0) var1[2:] returns everything after the second character (ie, it chops off the first two elements) var1[:3] returns everything up to the third character (ie, the first three elements) var1[1:-1] returns everything between the first and last character Steps: var1[::2] returns every other character starting with the first (0,2,4,6…) var1[::-1] returns the string backwards [aka Reversing a String] Concatenate: var1 = var1 + ‘ more text’ Multiply: var1*10 returns the var1 string 10 times Reverse: var1[::-1] (there is no built-in reverse function or method) Shift: var1[2:]+var1[:2] moves the first two characters to the end Built-in String Functions: len(string) returns the length of the string (including spaces) str(object) converts objects (int, float, etc.) into strings Built-in String Methods: .upper s.upper() returns a copy of the string converted to uppercase. .lower s.lower() returns a copy of the string converted to lowercase. .count s.count("string") adds up the number of times a character or sequence of characters appears in a string (case-sensitive!) NOTE: If s='hahahah' then s.count('hah') returns only 2. .isupper s.isupper() returns true if all cased characters in the string are uppercase. There must be at least one cased character. It returns false otherwise. .islower s.islower() returns true if all cased characters in the string are lowercase. There must be at least one cased character. It returns false otherwise. .find s.find(value,start,end) finds the index position of the first occurrence of a character/phrase in a range .replace s.replace("old","new") In Jupyter, hit Tab to see a list of available methods for that object. Hit Shift+Tab for more information on the selected method - equivalent to help(s.method) Splitting Strings: >>> greeting = 'Hello, how are you?' >>> greeting.split() returns ['Hello,', 'how', 'are', 'you?'] Note that the default delimiter is a space >>> fruit = 'Apple' >>> fruit.split('p') returns ['a', '', 'le'] (note the additional null value) >>> fruit.partition('p') returns ('a','p','ple') (head, sep, tail) Note also that methods work on objects, so 'The quick brown fox'.split() is valid Joining Strings: delimeter.join(list) joints a list of strings together, connected by a start string (delimeter) >>> list1 = ['Ready', 'aim', 'fire!'] >>> ', '.join(list1) returns 'Ready, aim, fire!'6 Turning Objects Into Strings : str() aka "casting objects as strings" >>> test = 3 >>> print('You have just completed test ' + str(test) + '.') You have just completed test 3. Escape characters: string = 'It's a nice day' returns an error string = 'It\'s a nice day' handles the embedded apostrophe \ can also break code up into multiline statements for clarity Note: embedded apostrophes are also handled by changing the apostrophe type string = "It's a nice day" is also valid. Placeholders: (%s, %f et al) Note: the .format() method is usually preferable. See below. Placeholders: %s acts as a placeholder for a string, %d for a number >>> print('Place my variable here: %s' %(string_name)) Note that %s converts whatever it's given into a string. print('Floating point number: %1.2f' %(13.145)) Floating point number: 13.14 where in 1.2, 1 is the minimum number of digits to return, and 2 is the number of digits to return past the decimal point. print('Floating point number: %11.4f' %(13.145)) Floating point number: 13.1450 There are 4 extra spaces (11 total characters incl decimal) NOTE: %s replicates the str() function, %r replicates the repr() function to do the same thing. Passing multiple objects: print('First: %s, Second: %s, Third: %s' %('hi','two',3)) First: hi, Second: two, Third: 3 Variables are passed in the order they appear in the tuple. Not very pythonic because to pass the same variable twice means repeating it in the tuple. Use .format instead (see below!) Omitting the argument at the end causes the placeholder to print explicitly: print('To round 15.45 to 15.5 use %1.1f') To round 15.45 to 15.5 use %1.1f …as does using %% (python sees this as a literal %) print('To round 15.45 to %1.1f use %%1.1f') %(15.45) To round 15.45 to 15.5 use %1.1f NOTE: Python 2.7 has a known issue when rounding float 5's (up/down seem arbitrary). See http://stackoverflow.com/questions/24852052/how-to-deal-with-the-ending-5-in-a-decimal-fraction-when-round-it For better performance, use the decimal module.7 FORMAT Double curly-brackets serve as positional placeholders and eliminate need for str() print('I prefer Python version {} to {}.'.format(3.4, 2.7)) Note the lack of quotes I prefer Python version 3.4 to 2.7. You can change the order of variables inside the function: print('I prefer Python version {1} to {0}.'.format(3.4, 2.7)) I prefer Python version 2.7 to 3.4. You can assign local variable names to placeholders: print('First: {x}, Second: {y}, Third: {z}.'.format(x=1., z='B', y=5)) First: 1.0, Second: 5, Third: B. Note that variables x, y and z are not defined outside of the function, and format handles the different object types. Unlike %s placeholders, format variables may be used more than once in a string, and stored in any order. Within the brackets you can assign field lengths, left/right alignments, rounding parameters and more print('{0:8} | {1:9}'.format('Fruit', 'Quantity')) print('{0:8} | {1:9}'.format('Apples', 3.)) print('{0:8} | {1:9}'.format('Oranges', 10)) Fruit | Quantity the 0 parameter takes the first object encountered Apples | 3.0 the 8 parameter sets the minimum field length to 8 characters Oranges | 10 By default, .format aligns text to the left, numbers to the right print('{0:8} | {1:<8}'.format('Fruit', 'Quantity')) print('{0:8} | {1:<8.2f}'.format('Apples', 3.66667)) print('{0:8} | {1:<8.2f}'.format('Oranges', 10)) Fruit | Quantity < sets a left-align (^ for center, > for right) Apples | 3.67 .2f converts the variable to a float with 2 decimal places Oranges | 10.00 You can assign field lengths as arguments: print('{:<{}} goal'.format('field', 9)) field goal With manual field specification this becomes {0:<{1}s} You can choose the padding character: print('{:-<9} goal'.format('field')) field---- goal You can truncate (the opposite of padding): …and by argument: print('{:.5}'.format('xylophone')) print('{:.{}}'.format('xylophone',7)) xylop xylopho Conversion tags enable output in either str, repr, or (in python3) ascii: { !s} { !r} { !a} Format supports named placeholders (**kwargs), signed numbers, Getitem/Getattr, Datetime and custom objects. For more info: https://pyformat.info8 WORKING WITH LISTS: Built-in List Functions: del list1[1] removes the second item from the list len(list1) returns the number of objects in the list len(list1[-2]) returns the number of characters in the second-to-last string in the list, including spaces) Built-in List Methods: .append L.append(object) -- append object to end .count L.count(value) -> integer -- return number of occurrences of value .extend L.extend(iterable) -- extend list by appending elements from the iterable .index L.index(value, [start, [stop]]) -> integer -- return first index of value. Raises ValueError if the value is not present. .insert L.insert(index, object) -- insert object before index .pop L.pop([index]) -> item – remove and return item at index (default last). .remove L.remove(value) -- remove first occurrence of a value. Raises ValueError if the value is not present. .reverse L.reverse() -- reverse *IN PLACE* .sort L.sort(cmp=None, key=None, reverse=False) -- stable sort *IN PLACE*; cmp(x, y) -> -1, 0, 1 In Jupyter, hit Tab to see a list of available methods for that object. Hit Shift+Tab for more information on the selected method - equivalent to help(l.method) Adding objects to a list: list1.append('rhubarb') Adding multiple objects to a list: list1.extend('turnips','squash') Adding contents of one list to another: list1.extend(list2) adds the contents of list2 to list1 NOTE: to add a list to another list as one object, use append. list1 = ['a', 'b'] list1.extend['c', 'd'] returns ['a', 'b', 'c', 'd'] list1.append['c', 'd'] returns ['a', 'b', ['c', 'd']] Inserting items into a list: list1.insert(3,'beets') puts ‘beets’ in the fourth position Sorting items in a list: list1.sort() rewrites the list in alphabetical order IN PLACE list2 = sorted(list1) creates a new list while retaining the original Reverse sorting a list: list1.sort(reverse=True) Reverse items in a list: list1.reverse() reverses the order of items in a list IN PLACE Remove items from a list: list1.pop() returns the last (-1) item and permanently removes it list1.pop(1) returns the second item and removes it You can capture the popped object: list3 = [1,2,3,4] x = list1.pop() print(x) returns 4 print(list3) returns [1,2,3] To check the existence of a value in a list: object in list returns True/False as appropriate (names work too) To join items use the string method: ' potato'.join(['one',', two','.']) returns 'one potato, two potato.'9 List Index Method list.index(object) returns the index position of the first occurrence of an object in a list. list1 = ['a','p','p','l','e'] list1.index('p') returns 1 Making a list of lists: list1=[1,2,3] list2=[4,5,6] list3=[7,8,9] matrix = [list1,list2,list3] matrix [[1, 2, 3], [4, 5, 6], [7, 8, 9]] Note: “matrix” absorbs the content of the lists, not the variable names. If you later change one of the lists, matrix will not be affected. Slicing: matrix[0] returns [1,2,3] matrix[0][0] returns 1 (the first object inside the first object) Reversing: matrix[1].reverse() returns [[1, 2, 3], [6, 5, 4], [7, 8, 9]] Slicing with a list comprehension: first_col = [row[0] for row in matrix] first_col returns [1,4,7] LIST COMPREHENSIONS [expression for item in iterable (if condition)] – always return a list allow you to perform for loops within one set of brackets Longhand: As a comprehension: l = [] l = [letter for letter in 'word'] for letter in 'word': print(l) l.append(letter) ['w', 'o', 'r', 'd'] print(l) ['w', 'o', 'r', 'd'] list_of_squares = [x**2 for x in range(6)] result: [0, 1, 4, 9, 16, 25] even_numbers = [num for num in range(7) if num%2==0] result: [0, 2, 4, 6] Convert Celsius to Fahrenheit: celsius = [0,10,20.1,34.5] fahrenheit = [(temp*(9/5)+32) for temp in celsius] type 9/5.0 in Python 2! result: [32.0, 50.0, 68.18, 94.1] Nested list comprehensions: fourth_power = [x**2 for x in [x**2 for x in range(6)]] result: [0, 1, 16, 81, 256, 625]10 WORKING WITH TUPLES: Remember: Tuple elements cannot be modified once assigned (tuples are immutable) tuple1 = (1,2,3) max(tuple1) returns 3, min(tuple1) returns 1 Note: commas define tuples, not parentheses. hank = 1,2 assigns the tuple (1,2) to hank Built-in Tuple Methods: (there are only 2) .count T.count(value) -> integer -- return number of occurrences of value .index T.index(value, [start, [stop]]) -> integer -- return first index of value. Raises ValueError if the value is not present. WORKING WITH DICTIONARIES: dict1 = {'Tom':4, 'Dick':7, 'Harry':23} To update a value: dict1['Harry'] = 25 To increase a value: dict1['Harry'] += 100 (the pythonic way to add/subtract/etc. value) To clear a dictionary: dict1.clear() (keeps the dictionary, but now it’s empty of values) To delete a dictionary: del dict1 dict1.keys() returns ['Dick', 'Tom', 'Harry'] dict1.values() returns [7,4,23] NOTE: Dictionaries are unordered objects! dict1.items() returns [('Dick',7),('Tom',4),('Harry',23)] a list of tuples! To add one dictionary to another: dict1.update(dict2) Nesting dictionaries: dict3 = {'topkey':{'nestkey':{'subkey':'fred'}}} dict3['topkey']['nestkey']['subkey'].upper() returns 'FRED' Dictionary Comprehensions: {key:value for key,value in iterable} used to create a dictionary {key:value for value,key in iterable} used if x,y appear in y,x order in iterable WORKING WITH SETS: To declare an empty set: set1=set() (because set1={} creates an empty dictionary) A list can be cast as a set to remove duplicates set([2,1,2,1,3,3,4]) returns {1,2,3,4} (items are put in order, though sets do not support indexing) A string can be cast as a set to isolate every character (case matters!) set('Monday 3:00am') returns {' ', '0', '3', ':', 'M', 'a', 'd', 'm', 'n', 'o', 'y'} A dictionary may use sets to store values (example of mixed drinks and their ingredients) Set Operators: a = {1,2,3} b = {3,4,5} c = {2,3} 1 in a returns True (set a contains a 1) Intersection & .intersection() a&b returns {3} Union | .union() a|b returns {1,2,3,4,5} Difference - .difference() a-b returns {1,2} items in a but not in b Exclusive ^ .symmetric_difference() a^b returns {1,2,4,5} items unique to each set Subset <= .issubset() c<=a returns True Proper subset < c= .issuperset() a>=a returns True Proper superset < a>a returns False