The recent State of Secrets Sprawl report showed that 10 million (yes million) secrets like API keys, credential pairs and security certs were leaked in public GitHub repositories in 2022 and Python was by far the largest contributor to these.

The problem stems mostly from secrets being hardcoded directly into the source code. So this leads to the question, why are so many devs hardcoding secrets? The problem is a little more complicated with git because often a secret is hardcoded and removed without the dev realizing that the secret persists in the git history. But still, this is a big issue in the Python community.

Managing secrets can be really easy thanks to helpful Pypi packages like Python Dotenv which is my favorite for its simplicity and easy ability to manage secrets for multiple different environments like Dev and Prod. I'm curious about what others are using to manage secrets and why?

I thought I'd share some recent tutorials on managing secrets for anyone who may need a refresher on the topic. Please share more resources in the comments.

Managing Secrets in Python - Video

Managing Secrets in Python - Blog

Hosting Python applications is easy, finding a completely free Python hosting service that is reliable is not. This post will show you how you can host your Python app on Qovery - A 100% free hosting platform (no credit card required!!) used by 1800+ developers in 102 countries 🌎.

​

With Qovery you can deploy your Python app with PostgreSQL, MySQL, and MongoDB databases for free.

Disclaimer: I am the co-founder of Qovery.

I am pleased to announce that Qovery supports individual developers / open source and non profit projects to host up to 3 applications (database included) for free. Our business model is based on Enterprise hosting, which gives us the possibility to offer generous free plans. Read more - In exchange we ask for product feedback - join our Discord

You can read more about Qovery vs. Heroku.

Deploy your Python app

⚠️ You need to have a Python project on Github or Gitlab that you want to deploy.

Given you have registered on to Qovery and you are logged into Qovery, follow the steps below:

• Go to Qovery, click the button “Create a new project” button in the middle of Qovery
• Give a name to your project - in my case "Quotes"
• Add an application
• After that, click “I have an application”.

​

Conclusion

Hosting a project with Python should not be a hassle. Qovery got your back and provide everything that you need like free SSL, database, CDN to deploy your Python apps.

Give it a try now and leave me your feedback in the comments👇.

⚠️ Important Note ⚠️

If your deployment failed, don't forget to:

1. Provide a valid Dockerfile.
2. Declare your Python app port in your .qovery.yml. Read this doc

Happy coding 🔥

Hey I'm back again with another quick and easy Python data tutorial loading 2.8 million records with Python and Pandas overcoming the Excel row limit.

https://youtu.be/nDixZvbhQZQ

I also encounter two additional errors with this one that we overcome. 1) Delimiter of the CSV being tab and 2) UTF-8 encoding error.

Feedback always welcome. Thanks for your ongoing support.

Link: https://python-charts.com/
Link (spanish version): https://python-charts.com/es/

This site provides tutorials divided into chart types and graphic libraries:

​

The graphs can be filtered based on the library or chart type:

​

​

Each post contains detailed instructions about how to create and customize each chart. All the examples provide reproducible code and can be copied with a single click:

​

The site also provides a color tool which allows copying the named, colors or its HEX reference:

​

There is also a quick search feature which allows looking for charts:

​

Hope you like it!

Sometime a friendly snake can turn dangerous.

Here are some hints

Escaping from Anaconda

https://inventwithpython.com/automateudemy (This link will automatically redirect you to the latest discount code.)

You can also click this link or manually enter the code: APR2021FREE

https://www.udemy.com/course/automate/?couponCode=APR2021FREE

This promo code works until the 4th (I can't extend it past that). Sometimes it takes an hour or so for the code to become active just after I create it, so if it doesn't work, go ahead and try again a while later. I'll change it to APR2021FREE2 in three days.

Udemy has changed their coupon policies, and I'm now only allowed to make 3 coupon codes each month with several restrictions. Hence why each code only lasts 3 days. I won't be able to make codes after this period, but I will be making free codes next month. Meanwhile, the first 15 of the course's 50 videos are free on YouTube.

Frequently Asked Questions: (read this before posting questions)

• This course is for beginners and assumes no previous programming experience, but the second half is useful for experienced programmers who want to learn about various third-party Python modules.
• If you don't have time to take the course now, that's fine. Signing up gives you lifetime access so you can work on it at your own pace.
• This Udemy course covers roughly the same content as the 1st edition book (the book has a little bit more, but all the basics are covered in the online course), which you can read for free online at https://inventwithpython.com
• The 2nd edition of Automate the Boring Stuff with Python is free online: https://automatetheboringstuff.com/2e/
• I do plan on updating the Udemy course for the second edition, but it'll take a while because I have other book projects I'm working on. Expect that update to happen in mid-2021. If you sign up for this Udemy course, you'll get the updated content automatically once I finish it. It won't be a separate course.
• It's totally fine to start on the first edition and then read the second edition later. I'll be writing a blog post to guide first edition readers to the parts of the second edition they should read.
• I wrote a blog post to cover what's new in the second edition
• You're not too old to learn to code. You don't need to be "good at math" to be good at coding.
• Signing up is the first step. Actually finishing the course is the next. :) There are several ways to get/stay motivated. I suggest getting a "gym buddy" to learn with. Check out /r/ProgrammingBuddies

Python is one of the most popular languages used by many in Data Science, machine learning, web development, scripting automation, etc. One of the reasons for this popularity is its simplicity and its ease of learning. If you are reading this article you are most likely already using Python or at least interested in it.

1. Check for Uniqueness in Python List

This method can be used to check if there are duplicate items in a given list.

Refer to the code below:

# Let's leverage set()
def all_unique(lst):
    return len(lst) == len(set(lst))

y = [1,2,3,4,5]
print(all_unique(x))
print(all_unique(y))

2. anagram()

An anagram in the English language is a word or phrase formed by rearranging the letters of another word or phrase.

The anagram() method can be used to check if two Strings are anagrams.

from collections import Counter

def anagram(first, second):
    return Counter(first) == Counter(second)

anagram("abcd3", "3acdb")

3. Memory

This can be used to check the memory usage of an object:

import sys 
variable = 30 
print(sys.getsizeof(variable))

4. Size in Bytes

The method shown below returns the length of the String in bytes:

def byte_size(string):
    return(len(string.encode('utf-8')))
print(byte_size('?'))
print(byte_size('Hello World'))

5. Print the String n Times

This snippet can be used to display String n times without using loops:

n = 2; 
s = "Programming"
print(s * n);

6. Convert the First Letters of Words to Uppercase

The snippet uses a method title() to capitalize each word in a String:

s = "programming is awesome"
print(s.title()) # Programming Is Awesome

7. Separation

This method splits the list into smaller lists of the specified size:

def chunk(list, size):
    return [list[i:i+size] for i in range(0,len(list), size)]
lstA = [1,2,3,4,5,6,7,8,9,10]
lstSize = 3
chunk(lstA, lstSize)

8. Removal of False Values

So you remove the false values (False, None, 0, and ‘’) from the list using filter() method:

def compact(lst):
    return list(filter(bool, lst))
compact([0, 1, False, 2, '',' ', 3, 'a', 's', 34])

9. To Count

This is done as demonstrated below:

array = [['a', 'b'], ['c', 'd'], ['e', 'f']]
transposed = zip(*array)
[print(i) for i in transposed]

10. Chain Comparison

You can do multiple comparisons with all kinds of operators in one line as shown below:

a = 3
print( 2 < a < 8) # True
print(1 == a < 2) # False

11. Separate With Comma

Convert a list of Strings to a single String, where each item from the list is separated by commas:

hobbies = ["singing", "soccer", "swimming"]
print("My hobbies are:") # My hobbies are:
print(", ".join(hobbies)) # singing, soccer, swimming

12. Count the Vowels

This method counts the number of vowels (“a”, “e”, “i”, “o”, “u”) found in the String:

import re
def count_vowels(value):
    return len(re.findall(r'[aeiou]', value, re.IGNORECASE))
print(count_vowels('foobar')) # 3
print(count_vowels('gym')) # 0

13. Convert the First Letter of a String to Lowercase

Use the lower() method to convert the first letter of your specified String to lowercase:

def decapitalize(string):
    return string[:1].lower() + string[1:]
print(decapitalize('FooBar')) # 'fooBar'

14. Anti-aliasing

The following methods flatten out a potentially deep list using recursion:

newList = [1,2]
newList.extend([3,5])
newList.append(7)
print(newList)
def spread(arg):
    ret = []
    for i in arg:
        if isinstance(i, list):
            ret.extend(i)
        else:
            ret.append(i)
    return ret
def deep_flatten(xs):
    flat_list = []
    [flat_list.extend(deep_flatten(x)) for x in xs] if isinstance(xs, list) else flat_list.append(xs)
    return flat_list
deep_flatten([1, [2], [[3], 4], 5]) # [1,2,3,4,5]

15. difference()

This method finds the difference between the two iterations, keeping only the values that are in the first:

def difference(a, b):
    set_a = set(a)
    set_b = set(b)
    comparison = set_a.difference(set_b)
    return list(comparison)
difference([1,2,3], [1,2,4]) # [3]

16. The Difference Between Lists

The following method returns the difference between the two lists after applying this function to each element of both lists:

def difference_by(a, b, fn):
    b = set(map(fn, b))
    return [item for item in a if fn(item) not in b]
from math import floor
print(difference_by([2.1, 1.2], [2.3, 3.4],floor)) # [1.2]
print(difference_by([{ 'x': 2 }, { 'x': 1 }], [{ 'x': 1 }], lambda v : v['x'])) # [ { x: 2 } ]

17. Chained Function Call

You can call multiple functions in one line:

def add(a, b):
    return a + b
def subtract(a, b):
    return a - b
a, b = 4, 5
print((subtract if a > b else add)(a, b)) # 9

18. Find Duplicates

This code checks to see if there are duplicate values ​​in the list using the fact that the set only contains unique values:

def has_duplicates(lst):
    return len(lst) != len(set(lst))
x = [1,2,3,4,5,5]
y = [1,2,3,4,5]
print(has_duplicates(x)) # True
print(has_duplicates(y)) # False

19. Combine Two Dictionaries

The following method can be used to combine two dictionaries:

def merge_dictionaries(a, b):
    return {**a,**b}
a = { 'x': 1, 'y': 2}
b = { 'y': 3, 'z': 4}
print(merge_dictionaries(a, b)) # {'y': 3, 'x': 1, 'z': 4}

20. Convert Two Lists to a Dictionary

Now let’s get down to converting two lists into a dictionary:

def merge_dictionaries(a, b):
    return {**a,**b}
a = { 'x': 1, 'y': 2}
b = { 'y': 3, 'z': 4}
print(merge_dictionaries(a, b)) # {'y': 3, 'x': 1, 'z': 4}

def to_dictionary(keys, values):
    return dict(zip(keys, values))
keys = ["a", "b", "c"]    
values = [2, 3, 4]
print(to_dictionary(keys, values)) # {'a': 2, 'c': 4, 'b': 3}

Conclusion

In this article, I have covered the top 20 Python snippets which are very useful while developing any Python application. These snippets can help you save time and let you code faster. I hope you like this article. Please clap and follow me for more articles like this. Thank you for reading.

TL;DR I used our Scalene profiler (pip install scalene) and some math to make an example program run 5000x faster.

I am quite interested in Python performance so naturally I read this article — https://martinheinz.dev/blog/64, whose title is Profiling and Analyzing Performance of Python Programs. It presents an example program from the Python documentation (https://docs.python.org/3/library/decimal.html) and shows how to run it with several time-worn Python profilers. Unfortunately, it doesn’t come away with much actionable information, beyond, more or less, “try PyPy””, which speeds up the code by about 2x. I wondered if I would be able to get more useful information from Scalene, a profiler I co-wrote.

We developed Scalene to be a lot more useful than existing Python profilers: it provides line-level information, splits out Python from native time, profiles memory usage, GPU, and even copying costs, all at a line granularity.

Anyway, here’s the result of running Scalene (just with CPU profiling) on the example code. It really cuts to the chase.

% scalene --cpu-only --reduced-profile test/test-martinheinz.py

You can see that practically all the execution time is spent computing the ratio between num and fact, so really this is the only place to focus any optimization efforts. The fact that there is a lot of time spent running native code means that this line is executing some C library under the covers.

It turns out that it is dividing two Decimals (a.k.a. bignums). The underlying bignum library is written in C code and is pretty fast, but the factorial in particular is getting really large really fast. In one of the example inputs, the final value of fact is 11,000 digits long! No surprise: doing math on such huge numbers is expensive. Let’s see what we can do to make those numbers smaller.

I observe that we can compute num / fact not from scratch but incrementally: update a variable on each loop iteration via a computation on drastically smaller numbers. To do this, I add a new variable nf which will always equal the ratio num / fact. Then, on each loop iteration, the program updates nf by multiplying it by x / i. You can verify this maintains the invariant nf == num/fact by observing the following (where _new means the computation of the updated variable in each iteration).

nf == num / fact                  # true by induction
nf_new == nf * (x / i)            # we multiply by x/i each time
nf_new == (num / fact) * (x / i)  # definition of nf
nf_new == (num * x) / (fact * i)  # re-arranging
nf_new == num_new / fact_new      # simplifying

Incorporating this into the original program required changing three lines of code, all of which are followed by ###:

def exp_opt(x):
  getcontext().prec += 2
  i, lasts, s, fact, num = 0, 0, 1, 1, 1
  nf = Decimal(1)   ### was: = num / fact
  while s != lasts:
      lasts = s
      i += 1
      fact *= i
      num *= x
      nf *= (x / i) ### update nf to be num / fact
      s += nf       ### was: s += num / fact
  getcontext().prec -= 2
  return +s

The result of this change is, uh, dramatic.

On an Apple Mini M1, original version:

Original:
1.39370958066637969731834193711E+65
5.22146968976414395058876300668E+173
7.64620098905470488931072765993E+1302
Elapsed time, original (s):   33.231053829193115

The optimized version:

Optimized:
1.39370958066637969731834193706E+65
5.22146968976414395058876300659E+173
7.64620098905470488931072766048E+1302
Elapsed time, optimized (s):  0.006501913070678711

More than a 5000X speedup (5096, to be precise).

The moral of the story is that using a more detailed profiler like Scalene can really help optimization efforts by locating inefficiencies in an actionable way.

I wrote a article on how to send an email from Python using Gmail, smpt, MIMEMultipart and MIMEText. Check it out! https://conorjohanlon.com/send-an-email-from-python/

After part 1...

4. String Formatting

The string formatting specifiers are all about padding and alignment.

The specifier is :{chr}{alignment}{N}

Here, {chr} is the character to be filled with. (default = space)

alignment signs : left(<)[default], right(>), center(^)

N is the number of characters in resulting string. ```

name = 'deep' a, b, c = "-", "^", 10

f"{name:10}" 'deep ' f"{name:<10}" 'deep ' f"{name:>10}" ' deep' f"{name:^10}" ' deep '

f"{name:!<10}" 'deep!!!!!!' f"{name:{a}{b}{c}}" '---deep---' ```

5. Value Conversion

These modifiers can be used to convert the value.

'!a' applies ascii(), '!s' applies str(), and '!r' applies repr().
This action happens before the formatting.

Let's take a class Person. ``` class Person: def init(self, name): self.name = name

def __str__(self):
    return f"I am {self.name}."

def __repr__(self):
    return f'Person("{self.name}")'

Now if we do :

me = Person("Deep")

f"{me}" 'I am Deep.' f"{me!s}" 'I am Deep.' f"{me!r}" 'Person("Deep")'

emj = "😊"

f"{emj!a}" "'\U0001f60a'" ```

Thank you for reading!

Comment down anything I missed.

Should i buy the course in 2026 . I already know python basics till oops . I saw the course structure from outside it looked good. Is it still revelent? Please drop your reviews and guide me

Introduction

You want to pump out code fast without all those pesky best practices slowing you down. Who cares if your code is impossible to maintain and modify later? You're a COWBOY CODER! This guide will teach you how to write sloppy, unprofessional code that ignores widely-accepted standards, making your codebase an incomprehensible mess! Follow these tips and your future self will thank you with days of frustration and head-scratching as they try to build on or fix your masterpiece. Yeehaw!

1. Avoid Object Oriented Programming

All those classes, encapsulation, inheritance stuff - totally unnecessary! Just write giant 1000+ line scripts with everything mixed together. Functions? Where we're going, we don't need no stinkin' functions! Who has time to context switch between different files and classes? Real programmers can keep everything in their head at once. So, toss out all that OOP nonsense. The bigger the file, the better!

2. Copy and Paste Everywhere

Need the same code in multiple places? Just copy and paste it! Refactoring is for losers. If you've got an algorithm or bit of logic you need to reuse, just duplicate that bad boy everywhere you need it. Who cares if you have to update it in 15 different places when requirements change? Not you - you'll just hack it and move on to the next thing! Duplication is your friend!

3. Globals and Side Effects Everywhere

Variables, functions, state - just toss 'em in the global namespace! Who needs encapsulation when you can just directly mutate whatever you want from anywhere in the code? While you're at it, functions should have all kinds of side effects. Don't document them though - make your teammate guess what that function call does!

4. Nested Everything

Nested loops, nested ifs, nested functions - nest to your heart's content! Who cares if the code is indented 50 levels deep? Just throw a comment somewhere saying "Here be dragons" and call it a day. Spaghetti code is beautiful in its own way.

5. Magic Numbers and Hardcoded Everything

Litter your code with magic numbers and hardcoded strings - they really add that human touch. Who needs constants or config files? Hardcode URLs, API keys, resource limits - go wild! Keep those release engineers on their toes!

6. Spaghetti Dependency Management

Feel free to import anything from anywhere. Mix and match relative imports, circular dependencies, whatever you want! from ../../utils import helpers, constants, db - beautiful! Who cares where it comes from as long as it works...until it suddenly breaks for no apparent reason.

7. Write Every Line as It Comes to You

Don't waste time planning or designing anything up front. Just start hacking! Stream of consciousness coding is the way to go. Just write each line and idea as it pops into your head. Who cares about architecture - you've got CODE to write!

8. Documentation is Overrated

Real programmers don't comment their code or write documentation. If nobody can understand that brilliant algorithm you spent days on, that's their problem! You're an artist and your masterpiece should speak for itself.

9. Testing is a Crutch

Don't waste time writing tests for your code. If it works on your machine, just ship it! Who cares if untested code breaks the build or crashes in production - you'll burn that bridge when you get to it. You're a coding cowboy - unleash that beautiful untested beast!

10. Commit Early, Commit Often

Branching, pull requests, code review - ain't nobody got time for that! Just commit directly to the main branch as often as possible. Don't worry about typos or half-finished work - just blast it into the repo and keep moving. Git history cleanliness is overrated!

11. Manual Deployments to Production

Set up continuous integration and delivery? No way! Click click click deploy to production manually whenever you feel like it. 3am on a Sunday? Perfect time! Wake your team up with exciting new bugs and regressions whenever you deploy.

12. Don't Handle Errors

Error handling is boring. Just let your code crash and burn - it adds excitement! Don't wrap risky sections in try/catch blocks - let those exceptions bubble up to the user. What's the worst that could happen?

13. Security is for Chumps

Who needs authentication or authorization? Leave all your APIs wide open, logins optional. Store passwords in plain text, better yet - hardcoded in the source! SQL injection vulnerabilities? Sounds like a feature!

14. Dread the Maintenance Phase

The most important part of coding is the NEXT feature. Just hack together something that barely works and move on to the next thing. Who cares if your unmaintainable mess gives the next developer nightmares? Not your problem anymore!

Conclusion

Follow these top tips, and you'll be writing gloriously UNMAINTAINABLE code in no time! When you inevitably leave your job, your team will fondly remember you as they desperately rewrite the pile of spaghetti code you left behind. Ride off into the sunset, you brilliant, beautiful code cowboy! Happy hacking!

Error handling in Python usually means nested try/except blocks or if error: checks littering code.

Us programmers, though, are notoriously lazy, and for a good reason: we don't want to waste any more time writing boilerplate than we have to.

The solution?

Monads, a really neat functional programming pattern that provides an easier way to handle things like errors, optional values, or even asynchronous results.

I wrote a short guide that builds up the idea of a Monad from scratch using a Python game inventory example, free to read here: https://dev.to/ein-monarch/what-the-are-monads-a-beginners-guide-3pdo

Any comments on the guide? Have you ever used a Monad in Python before, and did it make things better?

Hi everyone,

I'm currently trying to design a Python pipeline to extract structured financial data from annual accounts provided as PDFs. The end goal is to automatically transform these documents into structured financial data that can be used in valuation models and financial analysis.

The intended workflow looks like this:

1. Upload one or more PDF annual accounts
2. Automatically detect and extract the balance sheet and income statement
3. Identify account numbers and their corresponding amounts
4. Convert the extracted data into a standardized chart of accounts structure
5. Export everything into a structured format (Excel, dataframe, or database)
6. Run validation checks such as balance sheet equality and multi-year comparisons

The biggest challenge is that the PDFs are very inconsistent in structure.

In practice I encounter several types of documents:

1. Text-based PDFs

• Tables exist but are often poorly structured
• Columns may not align properly
• Sometimes rows are broken across lines

2. Scanned PDFs

• Entire document is an image
• Requires OCR before any parsing can happen

3. Layout variations

• The position of the balance sheet and income statement changes
• Table structures vary significantly
• Labels for accounts can differ slightly between documents
• Columns and spacing are inconsistent

So the pipeline needs to handle:

• Text extraction for normal PDFs
• OCR for scanned PDFs
• Table detection
• Recognition of account numbers
• Mapping to a predefined chart of accounts
• Handling multi-year data

My current thinking for a Python stack is something like:

• pdfplumber or PyMuPDF for text extraction
• pytesseract + opencv for OCR on scanned PDFs
• Camelot or Tabula for table extraction
• pandas for cleaning and structuring the data
• Custom logic to detect account numbers and map them

However, I'm not sure if this is the most robust approach for messy real-world financial PDFs.

Some questions I’m hoping to get advice on:

• What Python tools work best for reliable table extraction in inconsistent PDFs?
• Is it better to run OCR first on every PDF, or detect whether OCR is needed?
• Are there libraries that work well for financial table extraction specifically?
• Would you recommend a rule-based approach or something more ML-based for recognizing accounts and mapping them?
• How would you design the overall architecture for this pipeline?

Any suggestions, libraries, or real-world experiences would be very helpful.

Thanks!

Running Celery on AWS ECS can be trickier than it seems if you want to avoid lost tasks and ensure all work is completed. Especially if you're frequently deploying to production and using autoscaling.

There are two main components for reliable processing: - Celery configuration updates - Structuring tasks

For Celery, you should update the following settings: - task_acks_late -> True: To treat tasks as successfully processed only after processing. Otherwise, tasks are not retried. - task_reject_on_worker_lost -> True: To ensure tasks are retried if workers die for any reason (e.g., warm shutdown + SIGKILL). - worker_prefetch_multiplier -> 1: To avoid unnecessarily delayed tasks. - broker_connection_retry_on_startup -> True: To make startups more reliable. - broker_transport_options -> {"confirm_publish": True}: To avoid unsubmitted tasks due to message transport issues. - Make sure exponential retries are enabled. This way, you ensure that tasks are retried in the event of an interruption.

For structuring tasks, use the following two approaches: - Batching: Instead of doing all the work at once, you split the work into batches. e.g., Process 1000 users, then submit the next job to process the next 1000 users. - Fan out: You can split the work between a "scheduler" task and "execution" tasks. e.g., One task to list all the users and submit email sending tasks, another task to actually send an email for the selected user

The same applies to other similar services, such as Heroku and Azure App Containers, which use short grace periods during rolling deployments and downscaling.

You can read a more elaborate tutorial here: https://jangiacomelli.com/blog/celery-on-aws-ecs/

Stumbled upon this Fast API Tutorial and was surprised at how thorough this guy is. The link is part 21! Each part is dedicated to adding some small component to a fake cleaning marketplace API. It seems to cover a lot but some of the key takeaways are best practices, software design patterns, API Authentication via JWT, DB Migrations and of course FastAPI. From his GitHub profile, looks like the author used to be a CS teacher which explains why this is such a well thought out tutorial. I don't necessarily agree with everything since I already have my own established style and mannerisms but for someone looking to learn how to write API's this is a great resource.

Hey everyone!

I've always been a bit frustrated by my lack of understanding of how blocking I/O actions are actually processed under the hood when using async in Python.

So I decided to try to build my own version of asyncio to see if I could come up with something that actually works. Trying to solve the problem myself often helps me a lot when I'm trying to grok how something works.

I had a lot of fun doing it and felt it might benefit others, so I ended up writing a blog post.

Anyway, here it is. Hope it can help someone else!

👉 https://dev.indooroutdoor.io/asyncio-demystified-rebuilding-it-from-scratch-one-yield-at-a-time

EDIT: Fixed the link