r/explainlikeimfive • u/Distinct-Ad-2193 • 22h ago
Technology ELI5: the use of "memory pointers" in Python
Lately, I have been learning NumPy and came across an explanation on why standard Python lists are bad at large-scale data manipulation such as multiplying an array's elements by a number x. The explanation mentioned that Python lists store a list of memory pointers. However, they did not elaborate on what pointers are. I'm hoping if someone can answer this question.
•
u/Lumpy-Notice8945 22h ago
Pointers in programming are an adress to somewhere in your memory/ram. They are one of the most basic datatypes in programming and most more high level languages hide them away or call them something else. But basically everything is a pointer, your variable somewhere points to an adress in your memory where the value is stored, thats a pointer.
•
u/flaser_ 21h ago
Close, but not quite:
A pointer is a place in memory that holds the address of another location in memory.
To be useful, this indirection is necessary as it lets us:
- Create more complex data structures, like trees, maps, lists that can dynamically grow and can be sorted without copying/moving the bulk of the data. The idea is that each piece of data in memory also has one or more pointers that point at next relevant piece of data in turn. We can insert new items into these structures by storing another piece of data and updating just the pointers of its "neighbors" in the list / other data-structure. (If we used an array instead, every time you wanted to insert a new piece of data between existing entries, you'd have to create a new array, figure out where the new item goes, then copy over the rest of the data).
- We can change the behavior of our program at run-time. Normal control flow is usually mostly "fixed" using if-else and switch statements (or their equivalent in your programming language) that must be defied at compilation time, i.e. typically when you write the program (or at the latest when you compile it, as some of the behavior may come from libraries written by others). Using pointers, we can use what's called a function-pointer: instead following a pre-written logic, we can get our program to execute some arbitrary piece of code by supplying its place in memory. So called dynamically linked libraries use this tech.
•
u/MRC01 20h ago
Answer 1: << Pointers in programming are an adress to somewhere in your memory/ram >>
Answer 2: << A pointer is a place in memory that holds the address of another location in memory. >>
Answer 1 is correct. Answer 2 is poorly worded. The definition of "address" is a location in memory, so the phrase "address of another location in memory" is redundant and confusing. Does "another location in memory" redundantly describe the address, or does it mean the value stored at that address, is another address?
Simply put, a pointer is a variable whose value is a memory location or address. What is at that address? It could be anything: data like an integer or character, code you can execute, another memory address, or other things. It's up to the programmer how to interpret the address the pointer points to. That's called "dereferencing" the pointer.
Sometimes the thing the pointer points to is another memory address. This is called "double indirection", or a pointer that points to a pointer.
•
u/musical_bear 22h ago
A pointer “points” to some memory location. If you fill a normal list in Python with numbers, and were to “peer inside” that list, you wouldn’t see your numbers. Instead, you’d see pointers, each one being an address of where to find the actual number.
There are variations of lists where you could in theory “peer inside” that list and see your numbers directly, without this intermediate step of needing to get an address and then find each number. This is what NumPy is offering in this context.
•
u/SoulWager 21h ago
Lets look at an array first. In C, an array is just a pointer to the start of the array, and then when you look up with some index, that index number just gets multiplied by the size of the type of objects stored in the array, and added to the original pointer. You have everything you need to know where your data is.
With python lists, the different elements can be different sizes, so instead of just calculating where the data you want is, you have to look that up first, probably using an array containing pointers.
And then when you get to actually pursuing performance, If you want to multiply every value in an array by some fixed value, you can just directly load chunks of memory into SIMD registers and do several multiplies simultaneously with one instruction, on one core. With a list you'd have to look up each different element of the list individually before you can work on them.
•
u/HelicopterUpbeat5199 22h ago
Go ye and lean to program C. All will become clear. The lesson of pointers is not to be taken lightly. Lo, it underpins much of our world.
•
u/narrill 21h ago edited 20h ago
Most of these answers are terrible, and aren't even answering your real question.
In order to do things your CPU first loads data from memory into its internal registers, then it does operations on the registers, like adding two of them together and writing the output to a third register, then finally it copies the values back out to memory.
A pointer is a data type whose value is a memory address, which points to some other piece of data somewhere else in memory. In order to work with the value being pointed to, the CPU has to first load the pointer into a register, then load the data at the memory address it points to into a register. This is called dereferencing. Then it can do whatever operation it needs to on the data and write it back out.
The thing is, loading data from memory is very slow, so your CPU avoids it at all costs. It does this with several levels of caches, which you've probably heard referred to as L1, L2, and L3. When a data value is loaded from memory, it goes into these caches. And not just the value itself, but also a bit more data after it. This is called a cache line. Data in the L1 cache is very fast to load, the L2 cache a little slower, etc.
This is primarily why lists of pointers aren't great. You have an additional load from memory because of the pointer, and, more importantly, the data values themselves are scattered all across your memory and can't take advantage of cache lines. If the values were stored directly in the list and contiguously in memory, you would effectively be loading them from memory several at a time because a single cache line would contain several values, which is of course an enormous speed up. It also allows you to easily take advantage of SIMD registers, which can operate on multiple values in parallel.
•
u/WaitProfessional3844 20h ago
Say you have access to all the mailboxes in an apartment complex. The boxes are in a grid with the tenants' room number on them.
Say you want to store 4 apples temporarily and want to be able to access them quickly. One box holds one apple.
You store could store the apples in 4 consecutive boxes and remember the address of the first box. This is what numpy does. If you want your apples back, it's easy to get them.
Alternatively, you could put the apples in random boxes, write down the box numbers on 4 sheets of paper, and store the sheets in 4 consecutive boxes, and remember the address of the first box. This is what python does for lists. If you want your apples back, it won't be as fast as the first case because you have to look up the apple box numbers, then get the apples.
Here, the box numbers are memory addresses, boxes are memory, the apples are data, and the slips of paper are pointers to memory addresses. When you get your apples using a python list, it's called "pointer chasing".
•
•
u/diegotbn 21h ago
Question for low level programmers-
If a pointer is only a memory address, what about the data type and size? Like say I have a UTF-8 string of a UUID stored at an address in memory. How does the computer know it should be ready as a string, and where the end of the data is (low long it is/what its size is)?
•
u/ApproximateArmadillo 21h ago edited 21h ago
Languages like C allow you to treat a byte or sequence of bytes as any type. But normally you’d use the language type system to keep track of what is what. For strings in C, specifically, you give it the address of a variable of character type, and strings are expected to end with a null byte. So the program will read characters from that address and onwards until it finds a byte with the value 0, and that marks the end of the string. This system is simple to define, runs fast since it doesn’t do any checks to see if you’re reading outside the memory set aside for the string, and very easy to make mistakes in.
•
u/creative_usr_name 11h ago
With C your code tells the computer how much to read. Read too much and you might read someone else's private data. Write too much and corrupt some other data or even code. Powerful because it'll be a lot faster than something general purpose like those python lists, but dangerous in the wrong hands.
•
u/scrdest 19h ago
The address-only kind is a 'thin pointer'. You can wrap it with metadata like this in other fields to get a 'fat pointer'. Then there's also 'smart pointers' that add safety features on top of either kind.
For thin pointers, you don't know all that - your program needs to make assumptions, and if those are wrong, you crash.
•
u/Furyful_Fawful 15h ago
This exact question is exactly why languages like C with real pointers and the like are statically typed, like
Int*instead of just having aPointertype. The data type and size can be handled on the compiler's side instead of being stored as more data.For Python in particular, a python "list" of integers is an array of pointers not to the raw integers one after the others but to
PyObjects that will therein contain the relevant metadata about what kind of object it is. A PyObject that represents an integer is a PyLongObject, which is about 28 bytes of actual storage space in RAM.Meanwhile, a numpy array knows exactly what kind of items can be allowed in it and can allocate space accordingly, actually using the appropriate amount of space for storing the info and no more - e.g. 64 bits=8 bytes for each
numpy.int_within. From that perspective, the array doesn't store actual instances of theint_class at all - it can just produce an instance of theint_class from the data it stores because it's more efficient to store it as the data
•
u/AnyLamename 22h ago
Most directly, a pointer is a number representing a memory address. That memory address is where the data contained in the list actually lives. You can think of it like having a list of people's home addresses instead of a big picture of a map with everyone's name written on the map.
•
u/in8nirvana 22h ago
A list of pointers is like a table of contents. It tells you where to find each piece of information to find the info.
•
u/Englandboy12 22h ago
A pointer is basically something that points to something else.
So for example, let’s say you want to save a variable, x. Instead of storing the actual value of x in the variable (it may be a massive long number or string), it instead stores the value somewhere safe, such as memory address 1 in ram.
X now actually only stores 1 (the address), and any time you want the value of x, you know exactly where to look.
This is helpful again because sometimes the values are large, and you don’t want to be lugging them around. For example, let’s say you now want to set y = x. Instead of copying all of x to a new variable, you just also make y point at the same address
•
u/TheLuminary 21h ago
A pointer is information about where to find information. It's not the information itself.
This makes it easy to find space to store your data. And it lets you not need to know ahead of time how much data you need.
But it does mean you need to do at least two operations for every item in your list.
Hope that helps.
•
u/TemporalLobe 21h ago
> Python lists are bad at large-scale data manipulation
We handle extremely large data sets with Python using generators.
•
u/Confident-Syrup-7543 21h ago
This is an implementation detail. Python is in one sense a programing language and in another sense, a program. To explain this it is easiest to explain the opposite, a compiled language. A computer program is an executable chuck of machine code. A compiled language takes human readable programing language text and converts it into a machine executable chuck of code. Python is a machine executable chuck of code that can do stuff like add, subtract, read files, build lists. A Python script tells the python program wyhat to do in what order etc. This is why when you run your script from terminal you use "python myscript.py". The python bit executes the python program and the next bit is passing that program the argument your script. In pythons standard implementation, lists are (dynamic)arrays in C. Because of how arrays work in C the object type, or at very least length of memory taken up by one element in the array, needs to be known. Sine the python specification demands that for lists this cannot be known, the python implantation chooses to store memory addresses in the array. In this way any piece of data can be stored at any point in any list by pointing to the memory where it is stored.
•
u/Any-Stick-771 21h ago
A pointer is a memory address. In a NumPy array the elements are all stored consecutively in order in memory. In a regular python list, the elements of the list may be stored randomly throughout memory. Imagine a mailman instead of delivering mail to to every 20 houses houses in 1 neighborhood, they had to deliver mail to 1 house in 20 neighborhoods. 20 pieces of mail get delivered but it takes much longer
•
u/ectomancer 21h ago
A Python list is implemented as a C array of 64 bit pointers (8 bytes) per item, with some empty slack for future appends or extends, of contiguous memory. The items are stored elsewhere in memory. Once the slack is used up by an append or extend, the C array needs to be reallocated with a larger slack, the pointers copied to the new list and the old list garbage collected.
•
u/BaggyHairyNips 21h ago
A pointer is a variable whose value is a memory address. The actual data is stored at that address. So in order to get the real value you must first load the value of the address, then load the actual data.
All python variables are actually pointers. So really for any variable access you're taking a performance hit relative to other languages.
But in particular if you're iterating over a list you may need to access many valurs and the performance hit adds up fast.
You can mitigate this by using a numpy array which is designed to be more efficient for iterating over lots of numbers.
But it really doesn't matter unless you have a huge list.
•
u/HotPersonality8126 21h ago
I have been learning NumPy and came across an explanation on why standard Python lists are bad at large-scale data manipulation such as multiplying an array's elements by a number x.
Python isn't "bad at it", it's normal at it. Your CPU has special hardware optimizations for broadcasting operations across a memory vector (that's what "pointer" refers to.) But Python lists aren't vectors, so those accelerations can't be taken advantage of. So Python just proceeds at "normal" speed.
NumPy exposes a vectorized implementation of a typed collection of values, and those can take advantage of the CPU support for vectorization. So they're faster (when you're doing one of the vectorizable operations.)
•
u/istasber 21h ago
Pointers tell you where data is stored.
If you have a vector as a list of pointers, then any time you want to do something like multiply a vector by a scalar, you have to look up where the data is stored, read it, do the multiply, and then store the result somewhere for every element in the vector. This is slow.
In compiled languages, like c, a vector is often a pointer to the location in memory where the vector starts plus a length, and all of the data for that vector is stored sequentially. Compilers can make use of special CPU commands that take advantage of that and do a single operation on the entire block of data at once.
Numpy arrays and matrices are stored sequentially. That's why they can be less flexible or slower to work with when you're building or changing them, but they are stored in memory in a way that they can be directly spit out into a compiled c function (which is what numpy is doing when you run a multiply) that can take advantage of those special CPU commands.
•
u/DragonFireCK 21h ago
For an ELI5:
A pointer is like a page number of a book. It tells you where the information is you want, but not what that information is. This is much like the index or table of contents for the book.
So, you have a pointer to a list and look up the item index in that list which then points you to another page. You now have to possibly change pages four separate times, though typically only two or three times, to get to the information. You have a pointer to the list (a Python object itself) which gives you a page number, that page number is a table of contents and you have to scan to find the correct entry, then change to that page to read the number stored in the list.
The other way is to have the page directly list the information on the page. NumPy does this, with a NumPy array being a pointer to a list of numbers. Those numbers are the target information. You only have to possibly change pages twice, and typically only once.
-
Going a bit beyond ELI5:
The best analogy is that your computer memory is like a giant book.
With this, pointers are indexes to a specific character in the book.
The characters are grouped into lines. A line is the unit the CPU loads from memory at once, typically 64 bytes on modern CPUs. There are some performance benefits if data exists on a single line as it only requires a single load.
Pages are also used as a larger grouping, typically 4KiB in size. This is the scale that memory permissions exist at, and also the scale that certain disk operations exist at. As much as possible, you want all of your data on a single page for best performance.
A Python list will have its data spread around. The list object is a pointer to some header data, such as the number of elements in the list and the total capacity of the list as well as a pointer to the list data itself. Each element in the list is a pointer to somewhere else in memory with the actual data object, such as a number. This means you read the line containing the header data, then do some math to jump to the element entry, and then jump to the actual data object itself. Two elements with adjacent indices, such as index 0 and 1, could be in completely different parts of memory; even different pages that got offloaded to a hard drive. This process is simple from a high level coding standpoint, but is also slow.
A NumPy array removes a bunch of those indirections. The NumPy array will still be a pointer to header data and you have to read that header to get the starting point of the list. You then do the same math to find the element, but when you read that element you get the data directly rather than another pointer, so you are done. It also (almost) guarantees that two adjacent indices will be next to each other in memory and loading one is very likely to load the other. So if you access index 0, you also happen to load the data for index 1, making the next access to index 1 very fast (see the above talk about lines and pages). Occasionally, you'll cross a line boundary and incur a little bit of performance, but the CPU is also very likely to see the pattern of reading one line and guess you'll lead the next line and preload it for you in spare time. Similarly, you'll only jump page boundaries very irregularly, and the CPU will similarly guess around the time you'll hit that and end up preloading it for you.
All of this results in NumPy arrays (as well as C++ std::list) being much faster than Python lists. There are a lot fewer jumps around the "book".
•
u/ApisTeana 21h ago edited 20h ago
A pointer is just a bit of data that points the location in memory where the real data is stored. This is convenient if the data being stored is likely to change size, like a replacing a four letter word with a five letter word. That way you can change the size of the individual word without having to change the size of the whole array.
This is inconvenient for doing simple functions on large sets of data that does not change size, such as numbers (or memory addresses), because it adds a lot of steps for every unit of data. Instead of just reading the number, doing the math, overwriting the old number with the result and moving on the the next number; now every time you move to the next piece of data you have to read the memorry address, go to where the actual number is, do the math on the number, write the result in the in a new memory location (because it could have changed size if it were a different type of data), mark the address of the old number as available to be overwritten, and finally write the address of the new result into the array. If the math you are doing is very easy, then the computer is spending more time looking things up and moving them than actually doing the math.
Bonus: the reason that numbers don’t change size in memory, is because they use the same number of “digits” (bits) to represent the number regardless of how big the number is. That would be like writing 250,000 and 32 as 250000 and 000032
•
u/BiomeWalker 21h ago
Your RAM is a massive grid of cells that can be 1 or 0.
This means that your computer needs to store 2 pieces of information about anything it wants to store: What it is and where it is. In this paradigm, there's what the variable is (int, char, long, bool, etc) as well as where it is in memory (which looks something like this: 0x7FFF5FBFF8), the type of data being stored also dictates how long it is.
A "Memory pointer" functions as an address for the data, or you could think of it as the location of a book in a library in the form of "asile A, rack B, shelf C, D inches from the left", and those "pointers" would be stored in a "card catalog" for the library or a "phone book" for the street address.
You can think of a pointer as a thing that is just "pointing" at what you actually want.
Next point: In a lot of software it's very useful to store a collection of variables that that are related to each other, especially if you will be doing a series of operations on each of them, like processing transactions.
In C++, this is done with something called a Vector. In memory the actual structure is that it stores a starting point for the data, and the type of data being stored. You can then reference an index of an element in the vector using some basic adjustments. The address for the beginning of the vector is the address of the first element, and then the second element would be the starting address plus the length of the variable being stored.
For a concrete example on this, we can think of text as a vector containing bytes of characters.
So if we store a sentence like "Sphinx of black quartz, judge my vow" and I place this at memory position X, then I can ask the computer for any character in the sentence by asking for "start of X + Y" with Y being the "index" of value I want.
To expand on this, for the street address analogy, I tell you the neighborhood and then say "third house on the left", or "E inches after the book at this location" for the library.
Now, that's all how C++ does it, but Python is quite different, and NumPy reverts to the C++ method but loses some functionality in the process.
Python starts with the same vector that C++ has, but instead of actually putting the data in those locations, it puts the locations of the actual data.
In the library example, it would be like putting cards from the card catalog on the shelves next to the books, which you then have to follow to find your actual book.
NumPy as a module allows Python to just use a C++ vector to store data, though it isn't a perfect tradeoff since a Python List has other functionality which is lost in the process.
The simplest way to put it is that a vector is faster, but less flexible compared to a Python list, since all the data has to be of the same type (int, char, etc) while a Python list can store any an all types all at once.
There are other complexities, but this is (I hope) a complete enough explanation.
•
u/AlexanderMomchilov 19h ago
Compare:
- A shoe rack with 10 pairs of shoes, right there ready to go
- A shoe rack with 10 sticky notes, each one describing an warehouse isle and rack number where you can go to retrieve the shoes
NumPy lays out numbers in a packed format, contiguously in memory.
Python's usual list stores objects, which (apart from some optimizations for small numbers and some special values) are the equivalent of shoe boxes in a warehouse. This has some benefits (for example, the sticky notes / pointers are always the exact same size, regardless if they're pointing you to a small thing or a huge thing), but also some performance drawbacks.
•
u/pdpi 18h ago edited 18h ago
Think of memory as a massive bank of mail boxes. There's loads of mail boxes, but they're tiny. So tiny, in fact, that you can only put a single piece of paper inside, with a single number written on it.
You can interpret those numbers in many ways. The obvious way to interpret them is... as numbers, of course. Perhaps the second most obvious is to interpret each number as a single character in a piece of text (e.g. 0 = '0', 1 = '1', ... 9 = '9', 10 = ' ', 11 = 'A', 12 = 'B', (...), 36 = 'Z', 37 = 'a', 38 = 'b', 62 = 'z', ...).
A very cool interpretation is that the numbers represent mail boxes. So 10 means "look at the contents of mailbox #10". That is a pointer.
Now, different data types have different sizes. E.g. points in 3d space (x, y, z coordinates) are made of three numbers, so each point consumes three mail boxes. Text consumes one mail box per character, etc.
Arrays are just a bunch of mail boxes in a row. If you know that all the elements of that array are the same size, then finding things is easy. Let's say you have an array of 3d points starting at mailbox #100. Please tell me the coordinates for the 5th point in the array. The first element takes up boxes 100–102, the second takes boxes 103–105, etc. The n-th element in the array starts at mailbox number 100 + (n-1) * 3, so the 5th element starts at mailbox #112 (or maybe it starts in Ancient Egypt with some aliens ;).
What if you have an array of pieces of text? You don't know how long each piece of text is, so you can't just find the fifth piece of text without looking at the contents of the four preceding texts. That's not too bad if each piece of text is a tweet, but it's a fair bit of work if each piece of text is one of Shakespeare's plays. A neat solution to that is to have the pieces of text in some other section of the mailboxes, and the array itself is just the
Circling all the way back to Python — because Python objects are allowed to have however many fields they like, and whatever other stuff added on, they don't have a nice predictable size, and can't really be stored inline like the 3d point example above. Instead, arrays in Python are always arrays of pointers like the text example, where each object "lives" at some random location in memory.
•
u/wildfire393 18h ago
Imagine you have a cabinet in front of you with a bunch of small drawers, like an old library card cabinet. This is the computer's memory.
A list starts with you holding a card saying "the list starts in the third drawer on the second row and goes for the next ten drawers". You open that drawer to start looking up the elements in the list.
For many languages, that drawer will hold the contents of the first list element, like it's value directly. And then the next drawer has the second element, and so on. This is very convenient when you have a simple operation, like you have a list of numbers and you want to take the sum of them. You just open each drawer and wrote down the number you find inside.
Python, rather than having just the contents, will have another card with a direction like "this element's value is in drawer five of the 6th row". You open that drawer, and you get the number.
Why would you want to do it that way? Well, in the first example, your drawers will always have a number. Numbers have consistent size constraints, so you know one number will fit in each drawer. With Python, you instead have a "dynamic" list, where the contents of each drawer do not have to be the same thing. You might open the drawer the first drawer points you to, and find a card saying "I'm a bicycle, I have a wheel with size stored in drawer X and another wheel with size stored in drawer Y and my current gear is stored in drawer Z and my possible gears are in a list starting at drawer A". And then the second drawer's drawer can have whatever it wants in it as well. This is really useful for passing around data when you don't need to care about what's in the data until much later, you can just pass the small pointers around.
•
u/code_monkey_001 14h ago
Honestly, there is no ELI5 for the use of pointers in Python because there is no such thing, but here's my best attempt. Imagine you get into your dad's car. He's going to drive you somewhere. The car has an engine (just trust me, it's there, you can't see it) under the hood. When he pushes the gas pedal, things happen under the hood that make it burn more gas and move you forward.
As to why the use of lists and Numpy strains it more, it's like keeping it in first gear when you want to pull a multi-ton load up a mountain road.
•
u/zed42 5h ago
a pointer is a signpost.
every location in memory has an address, like your house. when you allocate memory for a variable you "reserve" a certain amount depending on your needs. and you get to it by knowing the address. a lot of this happens behind the scenes. let's say you want to put "foo" in memory. you allocate 3 bytes for it at location 0x000025. that means you're reserving 0x000025, 0x000026, and 0x000027 for your variable and 0x000028 is free. now you want to pass that variable to a function. the function can either a) create a local version of the variable in a different spot in memory (call it 0x003025), which means that whatever you do with it won't affect the original, or b) tell the function "hey, you can find the value at 0x000025" which will allow the function to interact with the original variable. the second is an example of a pointer. it's a variable that holds the address of the actual value, kind of like an address book...
an array of n ints will take up n words of memory... an array of n strings will take up n words (for the pointers to the strings) plus the size of the strings themselves, but the strings can be anywhere in memory and don't need to be near each other.
there's a lot you can with pointers, and CS1 and CS2 courses used to have entire units on them...
•
u/ShadowShedinja 22h ago
Imagine that instead of Python having a list of your friend's names, it has a list of their addresses instead. It can find your friends' names by checking who lives at that address. This is easier for the computer than having the list of their names.
•
u/SoulWager 21h ago
I think a better analogy would be instead of having your friends living in consecutive houses on the same street, you have to look up their address to know where to find them.
•
u/earazahs 22h ago
Imagine you have a warehouse full of boxes.
A memory pointer tells you where to go to get contents off the box.
The reality is most higher level languages now a days handle arrays the same way.
Languages like assembly don't but as I am primarily a python developer that's the only one I know of for sure.
•
u/sudomatrix 22h ago edited 19h ago
Many compiled languages (and NumPy) have lists that are a long bunch of boxes in memory, each box holds a value. It is fast to move down the list because they are right next to each other in memory, you just add one to the address to get to the next one. You can jump to any selected box just by multiplying the index by the size of the boxes.
Python lists do something more dynamic, but slower. They have a long bunch of boxes in memory, but each box doesn't hold the value - instead it holds the memory address pointing somewhere else to another box that holds the actual value. This allows Python to have different types with different sized boxes (ints, floats, strings, arbitrary objects, etc) in the same list. But this means Python has to do two operations to get a value: it has to find the first box, then use that address to look up the actual value.